CN115188048A - Sight line correction method and device, electronic equipment and storage medium - Google Patents
Sight line correction method and device, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN115188048A CN115188048A CN202210807431.3A CN202210807431A CN115188048A CN 115188048 A CN115188048 A CN 115188048A CN 202210807431 A CN202210807431 A CN 202210807431A CN 115188048 A CN115188048 A CN 115188048A
- Authority
- CN
- China
- Prior art keywords
- sight line
- coordinate
- face image
- sight
- estimation
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/16—Human faces, e.g. facial parts, sketches or expressions
- G06V40/168—Feature extraction; Face representation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/20—Image preprocessing
- G06V10/22—Image preprocessing by selection of a specific region containing or referencing a pattern; Locating or processing of specific regions to guide the detection or recognition
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/20—Image preprocessing
- G06V10/24—Aligning, centring, orientation detection or correction of the image
Abstract
The embodiment of the application discloses a sight line correction method, a sight line correction device, electronic equipment and a storage medium, wherein the method comprises the following steps: when the electronic equipment is in a target use scene, acquiring a first face image of a target user through a camera module of the electronic equipment; obtaining a first sight estimation coordinate corresponding to the first face image, wherein the first sight estimation coordinate is obtained by analyzing the first face image through a sight estimation model; and determining a sight line correction parameter according to the first sight line estimation coordinate and the sight line real coordinate corresponding to the first face image, wherein the sight line correction parameter is used for correcting the sight line estimation coordinate output by the sight line estimation model. By implementing the method, the sight line correction parameters with good correction effect can be determined.
Description
Technical Field
The present disclosure relates to the field of gaze tracking technologies, and in particular, to a gaze correction method and apparatus, an electronic device, and a storage medium.
Background
At present, in order to improve the accuracy of sight estimation, the existing sight estimation system mostly uses sight correction parameters to correct sight estimation coordinates output by a sight estimation model. In practice, it is found that the existing sight line correction method often needs equipment to specially guide a user to stare at output special signs in sequence, and if the user is not well matched, the correction effect of the determined sight line correction parameters on the sight line estimation coordinates is often poor.
Disclosure of Invention
The embodiment of the application provides a sight line correction method and device, electronic equipment and a storage medium, and sight line correction parameters with good correction effects can be determined.
A first aspect of an embodiment of the present application provides a gaze correction method, including:
when the electronic equipment is in a target use scene, acquiring a first face image of a target user through a camera module of the electronic equipment;
obtaining a first sight line estimation coordinate corresponding to the first face image, wherein the first sight line estimation coordinate is obtained by analyzing the first face image through a sight line estimation model;
and determining a sight line correction parameter according to the first sight line estimation coordinate and the sight line real coordinate corresponding to the first face image, wherein the sight line correction parameter is used for correcting the sight line estimation coordinate output by the sight line estimation model.
A second aspect of the embodiments of the present application provides a device for determining a gaze correction parameter, including:
the system comprises an image acquisition unit, a display unit and a control unit, wherein the image acquisition unit is used for acquiring a first face image of a target user through a camera module of the electronic equipment when the electronic equipment is in a target use scene;
a sight line coordinate estimation unit, configured to obtain a first sight line estimation coordinate corresponding to the first face image, where the first sight line estimation coordinate is obtained by analyzing the first face image through a sight line estimation model;
and the correction parameter determining unit is used for determining a sight line correction parameter according to the first sight line estimation coordinate and the sight line real coordinate corresponding to the first face image, wherein the sight line correction parameter is used for correcting the sight line estimation coordinate output by the sight line estimation model.
A third aspect of embodiments of the present application provides an electronic device, which may include:
a memory storing executable program code;
and a processor coupled to the memory;
the processor calls the executable program code stored in the memory, and when executed by the processor, the executable program code causes the processor to implement the method according to the first aspect of the embodiments of the present application.
A fourth aspect of the embodiments of the present application provides a computer-readable storage medium, on which executable program code is stored, and when the executable program code is executed by a processor, the computer-readable storage medium implements the method according to the first aspect of the embodiments of the present application.
A fifth aspect of the embodiments of the present application discloses a computer program product, which, when run on a computer, causes the computer to execute any one of the methods disclosed in the first aspect of the embodiments of the present application.
A sixth aspect of the present embodiment discloses an application publishing platform, configured to publish a computer program product, where when the computer program product runs on a computer, the computer is caused to execute any of the methods disclosed in the first aspect of the present embodiment.
According to the technical scheme, the embodiment of the application has the following advantages:
in the embodiment of the application, when the electronic equipment is in a target use scene, a first face image of a target user is acquired through a camera module of the electronic equipment; obtaining a first sight line estimation coordinate corresponding to the first face image, wherein the first sight line estimation coordinate is obtained by analyzing the first face image through a sight line estimation model; and determining a sight line correction parameter according to the first sight line estimation coordinate and the sight line real coordinate corresponding to the first face image, wherein the sight line correction parameter is used for correcting the sight line estimation coordinate output by the sight line estimation model.
According to the method, firstly, when the electronic equipment is in a target use scene, a first face image is collected through a camera module of the electronic equipment, then the first face image is analyzed through a sight estimation model, a first sight estimation coordinate corresponding to the first face image is obtained, and finally sight correction parameters for correcting the sight estimation coordinate output by the sight estimation model are determined according to the first sight estimation coordinate and a sight real coordinate corresponding to the first face image. Therefore, in the determining process of the sight line correction parameter disclosed by the embodiment of the application, the first face image is acquired in the daily use process of the user, sight line guide information does not need to be additionally output, the sight line correction parameter with a good correction effect can be determined, in addition, the determining process of the whole sight line correction parameter is insensitive to the user, and the convenience of determining the sight line correction parameter is further improved.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following briefly introduces the embodiments and the drawings used in the description of the prior art, and obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained according to the drawings.
Fig. 1 is a schematic view of a scene of a gaze correction method disclosed in an embodiment of the present application;
fig. 2 is a schematic flow chart of a sight line correction method disclosed in an embodiment of the present application;
fig. 3 is a schematic flow chart of another gaze correction method disclosed in the embodiments of the present application;
fig. 4 is a block diagram of a view line correcting apparatus according to an embodiment of the present application;
fig. 5 is a block diagram of an electronic device according to an embodiment of the present disclosure.
Detailed Description
The embodiment of the application provides a sight line correction method and device, electronic equipment and a storage medium, which can improve the correction effect of sight line correction parameters.
For a person skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only some embodiments of the present application, and not all embodiments. The embodiments in the present application shall fall within the protection scope of the present application.
It is understood that the electronic device involved in the embodiments of the present application may include a general electronic user terminal with a screen, such as a mobile phone, a smart phone, a portable terminal, a Personal Digital Assistant (PDA), a Portable Multimedia Player (PMP) device, a notebook Computer, a notebook (Note Pad), a Wireless Broadband (Wibro) terminal, a tablet Computer (PC), a smart PC, a point of sale (POS), a car Computer, and the like.
The electronic device may also include a wearable device. The wearable device may be worn directly on the user or may be a portable electronic device integrated into the user's clothing or accessory. The wearable device is not only a hardware device, but also can realize powerful intelligent functions through software support, data interaction and cloud server interaction, such as: the system has the functions of calculation, positioning and alarming, and can be connected with a mobile phone and various terminals. Wearable devices may include, but are not limited to, wrist-supported watch types (e.g., wrist watches, wrist-supported products), foot-supported shoes types (e.g., shoes, socks, or other leg-worn products), head-supported Glass types (e.g., glasses, helmets, headbands, etc.), and various types of non-mainstream products such as smart clothing, bags, crutches, accessories, and the like.
The sight line estimation refers to judging the position of the eye gaze of a person through a picture of the face of the person. The sight line estimation has very wide application in the fields of intelligent household appliances, intelligent computers, virtual games, automobile driving and military affairs. Taking driving of an automobile as an example, a camera is mounted in front of a windshield of the automobile or in the center of a steering wheel, a picture of a driver is taken, and then where a user is looking is judged according to the picture, which is the sight estimation in the driving scene of the automobile. The sight line estimation in the automobile driving scene can analyze whether the sight line of a driver is normal or not, so that the driving safety monitoring is carried out.
At present, in order to improve the accuracy of sight estimation, most of the existing sight estimation systems use sight correction parameters to correct sight estimation coordinates output by a sight estimation model. In practice, it is found that the existing sight line correction method often needs equipment to specially guide a user to stare at output special signs in sequence, and if the user is not well matched, the correction effect of the determined sight line correction parameters on the sight line estimation coordinates is often poor.
The method aims to determine the sight line correction parameters with good correction effect. The specific sight line correction method is described in detail below with reference to the accompanying drawings.
Referring to fig. 1, fig. 1 is a schematic view of a scene of a gaze correction method disclosed in an embodiment of the present application. The scene schematic diagram shown in fig. 1 may include an electronic device 10, where the electronic device 10 may acquire a first face image of a target user through a camera module of the electronic device when the electronic device is in a target use scene, analyze the first face image by using a line-of-sight estimation model to obtain a first line-of-sight estimation coordinate corresponding to the first face image, and determine a line-of-sight correction parameter for correcting the line-of-sight estimation coordinate output by the line-of-sight estimation model according to the first line-of-sight estimation coordinate and a line-of-sight real coordinate corresponding to the first face image. Therefore, in the process of determining the sight line correction parameter by the electronic device 10, the first face image can be acquired in the daily use process of the user, the sight line correction parameter with a good correction effect can be determined without additionally outputting sight line guide information, and in addition, the determination process of the whole sight line correction parameter is insensitive to the user, so that the convenience of determining the sight line correction parameter is improved.
Referring to fig. 2, fig. 2 is a schematic flow chart of a gaze correction method disclosed in the embodiments of the present application. The sight line correction method as shown in fig. 2 may include the steps of:
201. when the electronic equipment is in a target use scene, a first face image of a target user is acquired through a camera module of the electronic equipment.
The target usage scenario may refer to one or more scenarios in the normal usage process of the electronic device, and the embodiment of the present application is not limited. When the electronic equipment is in a target use scene, the sight line position of the user often has a certain rule, and different use scenes can be associated with the sight line position of the user. It is understood that the first face image may be collected unknowingly during the daily use of the electronic device by the user, without requiring the electronic device to output additional steps of correction information, thereby providing an implicit gaze correction method.
In some embodiments, the number of frames of the first face image may be one or more frames, and the embodiments of the present application are not limited. In the case where the number of frames of the first face image is multiple frames, the multiple frames of the first face image may be acquired under the same or different target usage scenarios. For example, the acquisition period of an image in a target usage scenario may be 100 milliseconds.
In some embodiments, in a case where the number of frames of the first face image is plural, the number of frames of the first face image may be greater than or equal to a frame number threshold, which is associated with the number of correction parameters of the sight line correction parameter, the greater the number of correction parameters, the greater the frame number threshold. Illustratively, the frame number threshold is 5 frames.
The following describes a target usage scenario by way of example:
illustratively, the electronic device comprises a vehicle-mounted terminal, and the target usage scenario may include any one or a combination of the following: the control system comprises a central console and the like, wherein the central console is used for controlling left turning of a vehicle, controlling right turning of the vehicle, controlling the vehicle to output speed warning information, controlling the vehicle to output oil quantity warning information, controlling vehicle dumping (comprising a left dumping warehouse and a right dumping warehouse), and controlling the vehicle. When the vehicle is controlled to turn left, the user can see the left view mirror of the vehicle with high probability; when the vehicle is controlled to turn right, the user can see the right view mirror of the vehicle with high probability; when the vehicle is controlled to output speed warning information and oil quantity warning information, a user can see a vehicle instrument panel with high probability; when the vehicle is controlled to be left to turn over, the user can see the right view mirror with high probability; when the vehicle is controlled to be turned to the right, the user can see the left view mirror with high probability; when controlling the center console of the vehicle, the user will see the vehicle center console with a high probability.
Illustratively, the electronic device includes a user terminal (smartphone, wearable device, tablet computer, etc.), and the target usage scenario may include any one or a combination of the following: opening the front camera, displaying a popup window, displaying a password (such as a power-on password, a payment password, an authentication code and the like) input interface and the like. When the front camera is started, a user can see the front camera with high probability; when the popup is displayed, a user can see the display area of the popup with high probability; when the password input interface is displayed, the user can see the display area of the password input interface with high probability.
202. And obtaining a first sight line estimation coordinate corresponding to the first face image, wherein the first sight line estimation coordinate is obtained by analyzing the first face image through a sight line estimation model.
The sight line estimation model can be obtained through deep learning and has the capability of estimating sight line coordinates. The sight line estimation model can be obtained by training a large number of sample face images and sight line real coordinates corresponding to each sample face image.
The gaze estimation model is typically subject to some error. Illustratively, the line-of-sight estimation model has an error of less than or equal to 2 centimeters.
Aiming at different electronic equipment, the sight line estimation model can output sight line estimation coordinates under different coordinate systems. When the electronic device includes a vehicle-mounted terminal, the origin of the coordinate system of the first sight-line estimation coordinate may be the center of a steering wheel of the vehicle, the rightward direction (i.e., the passenger driving direction) is a positive direction in the x direction, and the downward direction (i.e., the underfoot direction) is a positive direction in the y direction.
When the electronic device includes a user terminal, the origin of the coordinate system of the first sight line estimation coordinate may be the upper left corner of the screen of the user terminal, the positive direction of x is towards the right, and the positive direction of y is towards the bottom.
In some embodiments, the gaze estimation model may or may not require image parameters of the image it analyzes, and the embodiments of the present application are not limited thereto. The image parameters may include, but are not limited to, a resolution of the image and/or a size of the image, etc. For example, in the case where the gaze estimation model requires image parameters of the image it analyzes, the gaze estimation model requires the image to be 320 × 320 in size and 640 × 640 in resolution.
In some embodiments, in a case that the number of frames of the first face image is multiple frames, analyzing the first face image by using the gaze estimation model to obtain a first gaze estimation coordinate corresponding to the first face image may include: and respectively analyzing each frame of first face image through a sight estimation model to obtain a first sight estimation coordinate corresponding to each frame of first face image.
In some embodiments, the process of analyzing each frame of the first face image by the gaze estimation model may include: the method comprises the following steps: the method comprises the steps of detecting a face area in each frame of first face image through a face detection model, extracting the face area from each frame of first face image, adjusting the extracted face area to a target size corresponding to a sight estimation model to obtain a target image corresponding to each frame of first face image, and analyzing the target image corresponding to each frame of first face image by using the sight estimation model to obtain a first sight estimation coordinate corresponding to each frame of first face image.
By implementing the method, before the first sight line estimation coordinate corresponding to the first face image is obtained, the face in the first face image is detected, and then the target image input to the sight line estimation model is determined based on the detected face region, so that the analysis of invalid image pixels can be reduced, the efficiency of obtaining the first sight line estimation coordinate corresponding to the first face image is improved, and the construction efficiency of sight line correction parameters is effectively improved.
203. And determining a sight correction parameter according to the first sight estimation coordinate and the sight real coordinate corresponding to the first face image, wherein the sight correction parameter is used for correcting the sight estimation coordinate output by the sight estimation model.
The sight line real coordinates corresponding to the multiple frames of first face images collected in the same target use scene are the same.
In some embodiments, a target usage scenario may correspond to one or more line-of-sight real coordinates, and the embodiments of the present application are not limited thereto.
The gaze real coordinate may represent any position in the gaze area of the target user, and the gaze real coordinate may represent a center position of the gaze area of the target user.
The following describes, by way of example, a case where one target usage scene corresponds to one sight line real coordinate:
example 1, when a vehicle is controlled to turn left, the real coordinate of the sight line corresponding to the first face image is the coordinate of the center point of the left view mirror of the vehicle; when the vehicle is controlled to turn right, the sight line real coordinate corresponding to the first face image is the coordinate of the central point of the right sight glass of the vehicle; when the vehicle is controlled to output speed warning information and oil quantity warning, the sight line real coordinate corresponding to the first face image is the center point coordinate of the vehicle instrument panel; when a vehicle is controlled to be left-reversed, the sight line real coordinate corresponding to the first face image is the coordinate of the central point of the right sight glass; when the vehicle is controlled to be inverted to the right, the sight line real coordinate corresponding to the first face image is the coordinate of the central point of the left sight glass; and when the center console of the vehicle is controlled, the sight line real coordinate corresponding to the first face image is the coordinate of the center point of the center console of the vehicle.
Example 2, when the front camera is turned on, the real coordinate of the sight line corresponding to the first face image is the coordinate of the center point of the front camera; when the pop-up window is displayed, the sight line real coordinate corresponding to the first face image is the center point coordinate of the display area of the pop-up window; when the password input interface is displayed, the sight line real coordinate corresponding to the first face image is the center point coordinate of the password input interface.
Under the condition that one target use scene corresponds to a plurality of sight line real coordinates, the sight line real coordinates matching the rotation condition can be determined from the plurality of sight line real coordinates corresponding to the target use scene by analyzing the rotation condition of the head of the user in the first face image.
For example, when the vehicle is controlled to be inverted to the right, the user may look at the left view mirror and may also look at the center console, and then the sight line real coordinate with the highest precision can be determined according to the rotation condition of the head of the user.
It should be noted that the coordinates of the center point of the left view mirror of the vehicle, the coordinates of the center point of the right view mirror of the vehicle, the coordinates of the center point of the instrument panel of the vehicle, and the coordinates of the center point of the center console of the vehicle need to be measured in advance. Similarly, the coordinates of the center point of the front camera, the center point of the display area of the popup window and the center point of the password input interface need to be measured in advance.
In some embodiments, the correction relation for the sight line estimation coordinates may be a linear relation based on a least square method. The correction relation can be expressed as: x1= a x0+ b y0+ c; y1= d x0+ e y0+ f. Where x1 denotes the abscissa of the true coordinates of the line of sight, y1 denotes the ordinate of the true coordinates of the line of sight, x0 denotes the abscissa of the estimated coordinates of the line of sight, and y0 denotes the ordinate of the estimated coordinates of the line of sight. a. b, c, d, e and f are all sight line correction parameters.
It should be noted that, knowing the first estimated coordinates of the line of sight and the real coordinates of the line of sight corresponding to each frame of the first face image, the first estimated coordinates of the line of sight and the real coordinates of the line of sight can be substituted into the correction relational expression to obtain a, b, c, d, and f, that is, the line of sight correction parameters can be determined.
In some embodiments, the multiple frames of first face images may be divided into multiple groups, and the first sight line estimated coordinates and the sight line real coordinates corresponding to each group of first face images are respectively substituted into the correction relation, so as to obtain multiple groups of correction parameters. For the first correction parameter (the first correction parameter is any one of the sight line correction parameters), a plurality of values corresponding to the first correction parameter may be taken from the plurality of sets of correction parameters, and a mode or an average of the plurality of values may be used as the value of the first correction parameter.
In some embodiments, the plurality of first face images of the same object use scene may be grouped into one group.
By implementing the method, firstly, the first face image is collected under the condition that the electronic equipment is in a target use scene, then the first face image is analyzed through the sight estimation model to obtain a first sight estimation coordinate corresponding to the first face image, and finally, the sight correction parameter for correcting the sight estimation coordinate output by the sight estimation model is determined according to the first sight estimation coordinate and the sight real coordinate corresponding to the first face image. Therefore, in the determining process of the sight line correction parameter disclosed by the embodiment of the application, the first face image is acquired in the daily use process of the user, sight line guide information does not need to be additionally output, the sight line correction parameter with a good correction effect can be determined, in addition, the determining process of the whole sight line correction parameter is insensitive to the user, and the convenience of determining the sight line correction parameter is further improved.
Referring to fig. 3, fig. 3 is a schematic flow chart of another gaze correction method disclosed in the embodiments of the present application. The sight line correction method as shown in fig. 3 may include the steps of:
301. when the electronic equipment is in a target use scene, a camera shooting module of the electronic equipment acquires a first face image of a target user.
302. And obtaining a first sight line estimation coordinate corresponding to the first face image, wherein the first sight line estimation coordinate is obtained by analyzing the first face image through a sight line estimation model.
For detailed descriptions of step 301 to step 302, please refer to the above description for step 201 to step 202, which is not described herein again.
303. And determining a sight correction parameter according to the first sight estimation coordinate and the sight real coordinate corresponding to the first face image, wherein the sight correction parameter is used for correcting the sight estimation coordinate output by the sight estimation model.
In some embodiments, in the case that the number of frames of the first face image is multiple frames, determining the line-of-sight correction parameter according to the first estimated line-of-sight coordinate and the line-of-sight real coordinate corresponding to the first face image may include: acquiring a first sight line estimation coordinate corresponding to each frame of first face image and a distance value between the first sight line estimation coordinate and a sight line real coordinate corresponding to each frame of first face image; taking the first face image with the distance value larger than the distance value threshold value as an invalid first face image, and taking the first face image with the distance value smaller than or equal to the distance value threshold value as an effective first face image; and determining sight line correction parameters according to the first sight line coordinate and the sight line real coordinate corresponding to each effective first face image.
In the embodiment of the present application, the distance value threshold may be summarized from a large amount of data. Illustratively, the distance value threshold is 10 centimeters. For any frame of the first face image, if the distance value between the corresponding first sight line estimated coordinate and the corresponding sight line real coordinate is greater than the distance value threshold, it is indicated that the sight line position of the target user is not at the preset sight line position. For example, if the first face image is captured while controlling the left turn of the vehicle, it indicates that the target user's gaze location may be high and may not fall on the vehicle's left view mirror. By implementing the method, the effectiveness of the first face image can be judged according to the distance value between the first sight line estimation coordinate and the corresponding sight line real coordinate, the invalid first face image is removed, the sight line correction parameter is determined only by using the first sight line estimation coordinate and the sight line real coordinate of the valid first face image, and the accuracy of the sight line correction parameter is improved.
In some embodiments, for any frame of the first face image, the calculation of the distance value between the first gaze estimation coordinate and the gaze real coordinate may include, but is not limited to, any of the following:
the method 1 includes calculating an absolute value of a horizontal coordinate difference value and an absolute value of a vertical coordinate difference value between the first sight line estimated coordinate and the sight line real coordinate, and accumulating the absolute values of the horizontal coordinate difference value and the vertical coordinate difference value to obtain a distance value between the first sight line estimated coordinate and the sight line real coordinate.
And 2, calculating the square of the horizontal coordinate difference value and the square of the vertical coordinate difference value between the first sight line estimation coordinate and the sight line real coordinate, and summing the square of the horizontal coordinate difference value and the square of the vertical coordinate difference value and opening the root to obtain the distance value between the first sight line estimation coordinate and the sight line real coordinate.
The calculation formula of the mode 1 can be expressed as: d = | x 1 -x 0 |+|y 1 -y 0 |；
The calculation formula of mode 2 can be expressed as:
wherein d represents a distance value between the first sight line estimated coordinate and the sight line real coordinate, x 1 Expressed as the abscissa, y, of the true coordinates of the line of sight 1 Expressed as the ordinate, x, of the true coordinate of the line of sight 0 Expressed as the abscissa, y, of the estimated coordinates of the first line of sight 0 The ordinate of the first sight line estimation coordinate is shown.
304. Obtaining a second sight estimation coordinate corresponding to a second face image of the first user, wherein the second sight estimation coordinate is obtained by analyzing the second face image through a sight estimation model; wherein the first user is the same as or different from the target user.
The manner of analyzing the second facial image through the line-of-sight estimation model may refer to the manner of analyzing the first facial image of the target user through the line-of-sight estimation model, and is not repeated here.
305. And correcting the second sight line estimation coordinate by using the sight line correction parameter to obtain a target sight line coordinate of the first user.
In some embodiments, the first user and the target user may be the same or different, and the embodiments of the present application are not limited.
In some embodiments, the gaze estimation model and gaze correction parameters may both be generic, applicable to any user.
In some embodiments, the gaze estimation model may be generic, and the gaze correction parameters may be specific to a particular user, which may further improve the correction effect. Illustratively, the error of the gaze estimation model is 2 cm, and since the condition of each eye is different, the gaze correction parameters for user a can reduce the error of the gaze estimation model, while for user b, the gaze correction parameters for user a do not necessarily reduce the error of the gaze estimation model.
Optionally, the gaze correction parameter determined in step 303 may be associated with identity information of the target user.
Further, correcting the second gaze estimation coordinate using the gaze correction parameter to obtain a target gaze coordinate of the first user may include: and when the identity information of the first user is matched with the identity information of the target user, correcting the second sight line estimation coordinate by using the sight line correction parameter to obtain the target sight line coordinate of the first user.
The identity information of the user (the first user, the target user) may include any one or a combination of the following: fingerprint information, voiceprint information, iris information, face information, and the like. The identity information of the first user is matched with the identity information of the target user, which may be that the identity information of the first user is the same as the identity information of the target user.
In some embodiments, when the electronic device includes an in-vehicle terminal, after step 305, if the target sight-line coordinate is not in the driving safety sight-line zone, warning information is output. When the target sight line coordinate is not in the driving safety sight line interval, the sight line of the first user is abnormal, and if the target sight line coordinate is not in the driving safety sight line interval, the sight line of the first user is normal. By implementing the method, safety warning can be performed when the sight of the driver is abnormal.
By implementing the method, on one hand, in the construction process of the sight line correction parameter, the first face image is collected in the daily use process of the user, sight line guide information does not need to be additionally output, the sight line correction parameter with a good correction effect can be determined, and in addition, the whole sight line correction parameter determination process is insensitive to the user, so that the convenience of determining the sight line correction parameter is improved. On the other hand, the sight line correction parameters can be applied to correct sight line estimation coordinates analyzed by the sight line estimation model, and the sight line estimation precision can be improved.
Referring to fig. 4, fig. 4 is a block diagram of a sight line correction device disclosed in the embodiment of the present application. The apparatus shown in fig. 4 may include an image acquisition unit 401, a sight line coordinate estimation unit 402, a correction parameter determination unit 403; wherein:
the image acquisition unit 401 is configured to acquire a first face image of a target user through a camera module of the electronic device when the electronic device is in a target use scene;
a sight line coordinate estimation unit 402, configured to obtain a first sight line estimation coordinate corresponding to the first face image, where the first sight line estimation coordinate is obtained by analyzing the first face image through a sight line estimation model;
a correction parameter determining unit 403, configured to determine a gaze correction parameter according to the first gaze estimation coordinate and the gaze real coordinate corresponding to the first face image, where the gaze correction parameter is used to correct the gaze estimation coordinate output by the gaze estimation model.
In some embodiments, the number of frames of the first face image is multiple frames, and the manner of determining the line-of-sight correction parameter by the correction parameter determining unit 403 is specifically that, according to the first estimated line-of-sight coordinate and the line-of-sight real coordinate corresponding to the first face image, the determining the line-of-sight correction parameter includes: a correction parameter determining unit 403, configured to obtain a first estimated sight line coordinate corresponding to each frame of the first face image, and a distance value between the first estimated sight line coordinate and a real sight line coordinate corresponding to each frame of the first face image; taking the first face image with the distance value larger than the distance value threshold value as an invalid first face image, and taking the first face image with the distance value smaller than or equal to the distance value threshold value as an effective first face image; and determining sight line correction parameters according to the first sight line coordinate and the sight line real coordinate corresponding to each effective first face image.
In some embodiments, the gaze coordinate estimation unit 402 is further configured to obtain second gaze estimation coordinates corresponding to a second facial image of the first user, the second gaze estimation coordinates being obtained by analyzing the second facial image through a gaze estimation model; wherein the first user is the same as or different from the target user;
further, the apparatus shown in fig. 4 may further include a gaze coordinate correction unit for correcting the second gaze estimation coordinate using the gaze correction parameter to obtain a target gaze coordinate of the first user.
In some embodiments, the correction parameter determining unit 403 is further configured to associate the gaze correction parameter with the identity information of the target user;
further, the manner in which the gaze coordinate correction unit is configured to correct the second gaze estimation coordinate using the gaze correction parameter to obtain the target gaze coordinate of the first user may specifically include: and the sight line coordinate correction unit is used for correcting the second sight line estimation coordinate by using the sight line correction parameter when the identity information of the first user is matched with the identity information of the target user so as to obtain the target sight line coordinate of the first user.
In some embodiments, the apparatus shown in fig. 4 may further include a safety warning unit, configured to output warning information if the target sight line coordinate is not within the driving safety sight line interval.
In some embodiments, the target usage scenario may include any one or a combination of: the control method comprises the steps of controlling the left turning of a vehicle, controlling the right turning of the vehicle, controlling the vehicle to output speed warning information, controlling the vehicle to output oil quantity warning information, controlling the vehicle to fall into a garage, controlling a center console of the vehicle, opening a front camera, displaying a pop-up window and displaying a password input interface.
Fig. 5 is a block diagram of an electronic device according to an embodiment of the present disclosure. The electronic device as shown in fig. 5 may include a processor 501, a memory 502 coupled to the processor 501, wherein the memory 502 may store one or more computer programs.
The Memory 502 may include a Random Access Memory (RAM) or a Read-Only Memory (ROM). The memory 502 may be used to store instructions, programs, code, sets of codes, or sets of instructions. The memory 502 may include a stored program area and a stored data area, wherein the stored program area may store instructions for implementing an operating system, instructions for implementing at least one function (such as a touch function, a sound playing function, an image playing function, etc.), instructions for implementing the various method embodiments described above, and the like. The storage data area may also store data created by the electronic device in use, and the like.
In the embodiment of the present application, the processor 501 further has the following functions:
when the electronic equipment is in a target use scene, acquiring a first face image of a target user through a camera module of the electronic equipment;
obtaining a first sight line estimation coordinate corresponding to the first face image, wherein the first sight line estimation coordinate is obtained by analyzing the first face image through a sight line estimation model;
and determining a sight correction parameter according to the first sight estimation coordinate and the sight real coordinate corresponding to the first face image, wherein the sight correction parameter is used for correcting the sight estimation coordinate output by the sight estimation model.
In the embodiment of the present application, the processor 501 further has the following functions:
acquiring a first sight line estimation coordinate corresponding to each frame of first face image and a distance value between the first sight line estimation coordinate and a sight line real coordinate corresponding to each frame of first face image;
taking the first face image with the distance value larger than the distance value threshold value as an invalid first face image, and taking the first face image with the distance value smaller than or equal to the distance value threshold value as an effective first face image;
and determining sight line correction parameters according to the first sight line coordinate and the sight line real coordinate corresponding to each effective first face image.
In the embodiment of the present application, the processor 501 further has the following functions:
obtaining a second sight estimation coordinate corresponding to a second face image of the first user, wherein the second sight estimation coordinate is obtained by analyzing the second face image through a sight estimation model; wherein the first user is the same as or different from the target user;
and correcting the second sight line estimation coordinate by using the sight line correction parameter to obtain a target sight line coordinate of the first user.
In the embodiment of the present application, the processor 501 further has the following functions:
correlating the sight line correction parameter with the identity information of the target user;
and when the identity information of the first user is matched with the identity information of the target user, correcting the second sight line estimation coordinate by using the sight line correction parameter to obtain the target sight line coordinate of the first user.
In the embodiment of the present application, the processor 501 further has the following functions:
and if the target sight line coordinate is not in the driving safety sight line interval, outputting warning information.
In the embodiment of the present application, the target usage scenario includes any one or a combination of the following: the control vehicle turns left, control vehicle turns right, control vehicle output speed warning information, control vehicle output oil mass warning information, control vehicle to fall the storehouse, control the central console of vehicle, open leading camera, show the bullet window and show password input interface.
The embodiment of the application discloses a computer-readable storage medium, which stores a computer program, wherein when the computer program is executed by a processor, the processor is enabled to realize part or all of the steps executed by the electronic device in the embodiment.
The embodiment of the application discloses a computer program product, which enables a computer to execute part or all of the steps executed by the electronic equipment in the embodiment when the computer program product runs on the computer.
The embodiment of the application discloses an application publishing platform, which is used for publishing a computer program product, wherein when the computer program product runs on a computer, the computer is enabled to execute part or all of the steps executed by the electronic device in the embodiment.
In the above embodiments, the implementation may be wholly or partially realized by software, hardware, firmware, or any combination thereof. When implemented in software, it may be implemented in whole or in part in the form of a computer program product.
The computer program product includes one or more computer instructions. The procedures or functions according to the embodiments of the present application are all or partially generated when the computer program instructions are loaded and executed on a computer. The computer may be a general purpose computer, a special purpose computer, a network of computers, or other programmable device. The computer instructions may be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another, e.g., the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. A computer-readable storage medium may be any available medium that a computer can store or a data storage device, such as a server, a data center, etc., that is integrated with one or more available media. The usable medium may be a magnetic medium (e.g., floppy Disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid State Disk (SSD)), among others.
It is clear to those skilled in the art that, for convenience and brevity of description, the specific working processes of the above-described systems, apparatuses and units may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.
In the several embodiments provided in the present application, it should be understood that the disclosed system, apparatus and method may be implemented in other manners. For example, the above-described apparatus embodiments are merely illustrative, and for example, a division of a unit is only a logical division, and other divisions may be realized in practice, for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may be in an electrical, mechanical or other form.
Units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.
In addition, functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.
The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present application may be substantially implemented or contributed to by the prior art, or all or part of the technical solution may be embodied in a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method of the embodiments of the present application. And the aforementioned storage medium includes: various media capable of storing program codes, such as a usb disk, a removable disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk.
The above embodiments are only used to illustrate the technical solutions of the present application, and not to limit the same; although the present application has been described in detail with reference to the foregoing embodiments, it should be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions in the embodiments of the present application.
Claims (9)
1. A gaze correction method, characterized in that the method comprises:
when the electronic equipment is in a target use scene, acquiring a first face image of a target user through a camera module of the electronic equipment;
obtaining a first sight estimation coordinate corresponding to the first face image, wherein the first sight estimation coordinate is obtained by analyzing the first face image through a sight estimation model;
and determining a sight line correction parameter according to the first sight line estimation coordinate and the sight line real coordinate corresponding to the first face image, wherein the sight line correction parameter is used for correcting the sight line estimation coordinate output by the sight line estimation model.
2. The method according to claim 1, wherein the number of frames of the first face image is multiple frames, and the determining of the gaze correction parameter according to the first gaze estimation coordinate and the gaze real coordinate corresponding to the first face image comprises:
acquiring a first sight line estimation coordinate corresponding to each frame of the first face image and a distance value between the first sight line estimation coordinate and a sight line real coordinate corresponding to each frame of the first face image;
taking the first face image with the distance value larger than the distance value threshold value as an invalid first face image, and taking the first face image with the distance value smaller than or equal to the distance value threshold value as an effective first face image;
and determining a sight line correction parameter according to the first sight line coordinate corresponding to each effective first face image and the sight line real coordinate.
3. The method according to claim 1 or 2, wherein after determining the gaze correction parameter according to the first gaze estimation coordinate and the gaze real coordinate corresponding to the first face image, the method further comprises:
obtaining a second sight estimation coordinate corresponding to a second face image of the first user, wherein the second sight estimation coordinate is obtained by analyzing the second face image through the sight estimation model; wherein the first user is the same as or different from the target user;
and correcting the second sight line estimation coordinate by using the sight line correction parameter to obtain a target sight line coordinate of the first user.
4. The method of claim 3, further comprising:
associating the gaze correction parameter with identity information of the target user;
the correcting the second gaze estimation coordinate using the gaze correction parameter to obtain a target gaze coordinate of the first user comprises:
and when the identity information of the first user is matched with the identity information of the target user, correcting the second sight line estimation coordinate by using the sight line correction parameter to obtain the target sight line coordinate of the first user.
5. The method of claim 3, wherein after the correcting the second gaze estimation coordinate using the gaze correction parameter to obtain the target gaze coordinate of the first user, the method further comprises:
and if the target sight line coordinate is not in the driving safety sight line interval, outputting warning information.
6. The method of claim 1, wherein the target usage scenario comprises any one or a combination of: the control vehicle turns left, controls the vehicle to turn right, controls the vehicle to output speed warning information, controls the vehicle to output oil quantity warning information, controls the vehicle to fall into a garage and controls a center console of the vehicle, opens a front camera, displays a popup window and displays a password input interface.
7. An apparatus for determining a gaze correction parameter, comprising:
the system comprises an image acquisition unit, a display unit and a control unit, wherein the image acquisition unit is used for acquiring a first face image of a target user through a camera module of the electronic equipment when the electronic equipment is in a target use scene;
a sight line coordinate estimation unit, configured to obtain a first sight line estimation coordinate corresponding to the first face image, where the first sight line estimation coordinate is obtained by analyzing the first face image through a sight line estimation model;
and the correction parameter determining unit is used for determining a sight line correction parameter according to the first sight line estimation coordinate and the sight line real coordinate corresponding to the first face image, wherein the sight line correction parameter is used for correcting the sight line estimation coordinate output by the sight line estimation model.
8. An electronic device, comprising:
a memory storing executable program code;
and a processor coupled to the memory;
the processor calls the executable program code stored in the memory, which when executed by the processor causes the processor to implement the method of any of claims 1-6.
9. A computer readable storage medium having executable program code stored thereon, wherein the executable program code, when executed by a processor, implements the method of any of claims 1-6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210807431.3A CN115188048A (en) | 2022-07-07 | 2022-07-07 | Sight line correction method and device, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210807431.3A CN115188048A (en) | 2022-07-07 | 2022-07-07 | Sight line correction method and device, electronic equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115188048A true CN115188048A (en) | 2022-10-14 |
Family
ID=83517643
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210807431.3A Pending CN115188048A (en) | 2022-07-07 | 2022-07-07 | Sight line correction method and device, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115188048A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117152411A (en) * | 2023-11-01 | 2023-12-01 | 安徽蔚来智驾科技有限公司 | Sight line calibration method, control device and storage medium |
-
2022
- 2022-07-07 CN CN202210807431.3A patent/CN115188048A/en active Pending
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117152411A (en) * | 2023-11-01 | 2023-12-01 | 安徽蔚来智驾科技有限公司 | Sight line calibration method, control device and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110058694B (en) | Sight tracking model training method, sight tracking method and sight tracking device | |
| CN110032271B (en) | Contrast adjusting device and method, virtual reality equipment and storage medium | |
| US10460164B2 (en) | Information processing apparatus, information processing method, eyewear terminal, and authentication system | |
| CN108198044B (en) | Commodity information display method, commodity information display device, commodity information display medium and electronic equipment | |
| US9497389B2 (en) | Information processing device, information processing method, and program for reduction of noise effects on a reference point | |
| EP3086275A1 (en) | Numerical value transfer method, terminal, cloud server, computer program and recording medium | |
| CN105247447A (en) | Systems and methods of eye tracking calibration | |
| WO2019020103A1 (en) | Target recognition method and apparatus, storage medium and electronic device | |
| CN107174824B (en) | Special effect information processing method and device, electronic equipment and storage medium | |
| CN111552368A (en) | Vehicle-mounted human-computer interaction method and vehicle-mounted equipment | |
| CN107422844B (en) | Information processing method and electronic equipment | |
| CN110969116B (en) | Gaze point position determining method and related device | |
| CN110780742B (en) | Eyeball tracking processing method and related device | |
| KR102287109B1 (en) | Method and device for correcting image processing area corresponding to skin | |
| US9619707B2 (en) | Gaze position estimation system, control method for gaze position estimation system, gaze position estimation device, control method for gaze position estimation device, program, and information storage medium | |
| CN109144250B (en) | Position adjusting method, device, equipment and storage medium | |
| CN110738078A (en) | face recognition method and terminal equipment | |
| US11450022B2 (en) | Voting space-based loop closure detection | |
| US11335090B2 (en) | Electronic device and method for providing function by using corneal image in electronic device | |
| CN111580665B (en) | Method and device for predicting fixation point, mobile terminal and storage medium | |
| CN115188048A (en) | Sight line correction method and device, electronic equipment and storage medium | |
| CN110568930A (en) | Method for calibrating fixation point and related equipment | |
| CN111027506B (en) | Method and device for determining sight direction, electronic equipment and storage medium | |
| CN112183173B (en) | Image processing method, device and storage medium | |
| CN110313006A (en) | A kind of facial image detection method and terminal device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |