JP2022517254A - Gaze area detection method, device, and electronic device - Google Patents

Abstract

The present invention provides a gaze area detection method, an apparatus, and an electronic device. The method obtains a face image collected in a predetermined three-dimensional space, executes line-of-sight detection based on the face image to obtain a line-of-sight detection result, and obtains a line-of-sight detection result with respect to the predetermined three-dimensional space. A pre-trained gaze area classifier is used to detect the type of target gaze area corresponding to the face image based on the gaze detection result, wherein the target gaze area is: It belongs to one of a plurality of defined gaze areas obtained by preliminarily dividing the predetermined three-dimensional space. [Selection diagram] Fig. 1

Description

<Mutual citation of related applications>
The present invention claims the priority of a Chinese patent application with a filing date of March 18, 2019, an application number of 2019102047793.1, and an invention title of "Gaze Area Detection Method, Device, and Electronic Device". However, the entire contents of the Chinese patent application are incorporated herein by reference.
The present invention relates to the field of computer vision technology, and more particularly to gaze area detection methods, devices, and electronic devices.

Gaze area detection can play an important role in applications such as intelligent driving, human-computer interaction, and security monitoring. Regarding human-computer interaction, the 3D position in the eye space is determined, and the 3D line-of-sight direction is combined to obtain the position of the human gaze point in the 3D space, which is output to the machine for further interactive processing. To do so. With regard to attention detection, by estimating the gaze direction of the eyes, it is possible to determine the gaze direction of the person, obtain the area of interest of the person, and determine whether or not the attention of the person is concentrated.

According to the first aspect of the present invention, a gaze area detection method is provided, in which the method obtains a face image collected in a predetermined three-dimensional space and executes line-of-sight detection based on the face image. By obtaining the line-of-sight detection result and using the gaze area classifier trained in advance for the predetermined three-dimensional space, the type of the target gaze area corresponding to the face image can be determined based on the line-of-sight detection result. Including detecting, where the target gaze area belongs to one of a plurality of defined gaze areas obtained by pre-dividing the predetermined three-dimensional space.

According to the second aspect of the present invention, a gaze area detection device is provided, in which the device is an image acquisition module for acquiring a face image collected in a predetermined three-dimensional space, and line-of-sight detection based on the face image. Using the line-of-sight detection module for obtaining the line-of-sight detection result and the gaze area classifier trained in advance for the predetermined three-dimensional space, the face image is created based on the line-of-sight detection result. It comprises a gaze area detection module for detecting the type of the corresponding target gaze area, wherein the target gaze area has a plurality of types of definitions obtained by preliminarily dividing the predetermined three-dimensional space. It belongs to one of the gaze areas.

According to a third aspect of the invention, a computer-readable recording medium in which a computer program is stored is provided so that when the computer program is executed by a processor, the processor realizes the method of the first aspect. do.

According to a fourth aspect of the present invention, an electronic device is provided, the electronic device includes a memory and a processor, and the computer program is stored in the memory, and the processor executes the computer program. In addition, the method of the first aspect described above is realized.

According to the embodiment of the present invention, it is necessary to train only the gaze area classifier corresponding to each three-dimensional space for the change of the predetermined three-dimensional space. Classifier training does not require large amounts of data and training speeds are faster, so the time cost and technical costs of transferring gaze area detection methods between different 3D spaces (eg, spaces of different vehicle models). Difficulties can be greatly reduced.

It is a flowchart of the gaze area detection method which concerns on an exemplary embodiment of this invention. It is a flowchart of the method of training the gaze area classifier for a predetermined 3D space which concerns on an exemplary embodiment of this invention in real time. It is a schematic diagram of a plurality of types of defined gaze areas according to an exemplary embodiment of the present invention. It is a flowchart of the method of determining the line-of-sight start point information of a person in a face image which concerns on an exemplary embodiment of this invention. It is a flowchart of the method of detecting the line-of-sight direction information of a person in a face image which concerns on an exemplary embodiment of this invention. It is a flowchart of the method of detecting the head appearance information of the person in the face image which concerns on an exemplary embodiment of this invention. It is a flowchart of the method of detecting the line-of-sight direction information of a person in a face image based on the head figure information which concerns on an exemplary embodiment of this invention. It is a flowchart of the method of obtaining the normalized face image by performing the normalization processing on the face image which concerns on an exemplary embodiment of this invention. It is a schematic diagram which performs the normalization processing on the acquired face image which concerns on an exemplary embodiment of this invention. It is a schematic diagram which outputs the type of the target gaze area by the classifier which concerns on the exemplary embodiment of this invention. It is a schematic diagram which outputs the name of the target gaze area by the classifier which concerns on the exemplary embodiment of this invention. It is a flowchart of the method of training the neural network for detecting the three-dimensional line-of-sight direction which concerns on an exemplary embodiment of this invention. It is a block diagram of the gaze area detection apparatus which concerns on an exemplary embodiment of this invention. It is a block diagram of the line-of-sight detection module of the gaze area detection device which concerns on an exemplary embodiment of this invention. It is a block diagram of another line-of-sight detection module of the gaze area detection device which concerns on an exemplary embodiment of this invention. 12 is a block diagram of an eye position detection submodule in FIGS. 12 and 13 according to an exemplary embodiment of the present invention. It is a block diagram of another line-of-sight detection module of the gaze area detection device which concerns on an exemplary embodiment of this invention. It is a block diagram of the appearance detection submodule of the line-of-sight detection module in FIG. 15 which concerns on an exemplary embodiment of this invention. It is a block diagram of the direction detection submodule of the line-of-sight detection module in FIG. 15 which concerns on an exemplary embodiment of this invention. It is a block diagram of the image processing unit of the direction detection submodule in FIG. 17 which concerns on an exemplary embodiment of this invention. It is a block diagram of another gaze area detection apparatus which concerns on an exemplary embodiment of this invention. It is a block diagram of another gaze area detection apparatus which concerns on an exemplary embodiment of this invention. It is a block diagram of another gaze area detection apparatus which concerns on an exemplary embodiment of this invention. It is a block diagram of another gaze area detection apparatus which concerns on an exemplary embodiment of this invention. It is a block diagram of the electronic device which concerns on an exemplary embodiment of this invention.

Here, exemplary embodiments will be described in detail and examples are shown in the drawings. Where the following description refers to drawings, the same numbers in different drawings indicate the same or similar elements, unless otherwise stated. The embodiments described in the following exemplary examples do not represent all embodiments consistent with the present invention. Conversely, they are merely examples of devices and methods consistent with some aspects of the invention described in the appended claims.

The terms used in the present invention are intended solely to illustrate specific embodiments and are not intended to limit the invention. The singular forms used in the present invention, such as "kind," "above," and "corresponding," are intended to include the plural unless the context clearly indicates other meanings. It should be understood that the term "and / or" as used herein refers to including any one or all possible combinations of one or more related listed items. ..

In the present invention, various information may be described using terms such as first, second, and third, but it should be understood that such information should not be limited by these terms. These terms are used only to distinguish the same type of information from each other. For example, without departing from the scope of the present disclosure, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein can be interpreted as "... if", "... then", or "in response to ...".

The present invention provides a gaze area detection method and can be applied to scenes such as intelligent driving, human-computer interaction, and security monitoring. The present invention will be described in detail with reference to an example of applying the gaze area detection method to an intelligent driving scene.

In the embodiments of the present invention, the executing entity involved may include a computer system and a camera provided in a predetermined three-dimensional space. A camera provided in a predetermined three-dimensional space can transmit the collected face image data of the user to the above-mentioned computer system. The computer system uses an artificial neural network to execute processing on the above facial image data to detect in which area of a predetermined three-dimensional space the user's attention is concentrated. That is, the target gaze area of the user can be detected, and the computer system responds, such as an instruction for driving a smart traveling vehicle based on the target gaze area of the user described above. Operation control information can be output.

The computer system may be a server, a server cluster, or a cloud platform, and may be a computer system in an electronic device such as a personal computer, an in-vehicle device, or a mobile terminal. The above-mentioned camera may be an in-vehicle device such as a camera in a driving recorder or a camera of a smart terminal. The smart terminal may include electronic devices such as smartphones, PDAs (Personal Digital Assistants), tablet computers, in-vehicle devices and the like. In the process of concretely realizing, the camera and the computer system may be independent of each other or connected to each other at the same time, and the gaze area detection method provided by the embodiment of the present invention can be jointly realized. .. Hereinafter, the gaze area detection method provided by the present invention will be described in detail with reference to an example of a computer system.

FIG. 1 is a flowchart of a gaze area detection method according to an exemplary embodiment of the present invention. The method can be performed by a computer system and can be applied to various smart devices (eg, smart transportation means, smart robots, smart home devices, etc.). As shown in FIG. 1, the method may include steps 11-13.

In step 11, a face image collected in a predetermined three-dimensional space is acquired.

Taking the example of the vehicle of the M model, the predetermined three-dimensional space is the space of the vehicle, and one camera can be fixedly installed in the internal space such as the position of the center console of the vehicle. The camera may collect a face image of a target target such as a driver and provide it to a computer system in real time or at a predetermined time cycle so that the computer system can acquire the collected face image. can.

In step 12, the line-of-sight detection is executed based on the face image, and the line-of-sight detection result is obtained.

In the embodiment of the present invention, the computer system can execute the line-of-sight detection based on the above-mentioned face image and obtain the line-of-sight detection result. The line-of-sight detection is to obtain the line-of-sight detection result by analyzing the position and / or the line-of-sight direction of the eyes in the facial image. The present invention is not limited to the method of performing the line-of-sight detection, that is, the method referred to in the embodiment of the present invention may be adopted to perform the line-of-sight detection, or other conventional methods may be adopted. Then, the line-of-sight detection may be executed. The above-mentioned line-of-sight detection result may include line-of-sight start point information and line-of-sight direction information of a person in a face image, and may further include information such as a head figure of a person in a face image.

In step 13, the type of the target gaze area corresponding to the face image is detected based on the gaze detection result by using the gaze area classifier trained in advance for the predetermined three-dimensional space.

The target gaze area belongs to one of a plurality of defined gaze areas obtained by preliminarily dividing the predetermined three-dimensional space. For example, each space that the driver can gaze at during the vehicle traveling process, such as a windshield, a rear-view mirror, or another space in the vehicle, can be set as a predetermined three-dimensional space.

As in the above example, the computer system obtains the line-of-sight detection result of the person in the face image, and then gazes at the above-mentioned line-of-sight detection result of the above-mentioned M model intelligent driving vehicle trained in advance. By inputting into the area classifier, it is possible to detect the type of target gaze area corresponding to the above facial image, i.e., which area of the vehicle the person in the facial image, such as the driver when collecting the image. It is possible to detect whether or not you are watching.

In the present invention, the gaze area classifier for the predetermined three-dimensional space is pre-trained by a computer system based on the training sample set for the predetermined three-dimensional space, and here, the training sample. The set includes a plurality of gaze feature samples, and each gaze feature sample contains gaze start point information, gaze direction information, and labeling information of the gaze area type corresponding to the gaze feature sample, and is a labeled gaze area. The type belongs to one of a plurality of defined gaze areas obtained by dividing the predetermined three-dimensional space.

According to an embodiment of the present invention, before training the gaze area classifier for a predetermined 3D space, for a 3D space area in the predetermined 3D space where the line of sight of the eye may be noticed. And finely categorize to obtain multiple types of defined gaze areas, and perform classifier training based on a training sample set corresponding to multiple types of defined gaze areas for a given 3D space. Obtain a gaze area classifier. Subsequently, the target gaze area information can be accurately detected based on the line-of-sight detection result using the gaze area classifier, the calculation is easy, and the misjudgment rate of the target gaze area is effectively reduced. More accurate information can be provided for subsequent operations.

The line-of-sight detection step corresponding to the above step 12 has nothing to do with the distribution of a plurality of types of defined gaze areas in a predetermined three-dimensional space, and the gaze area detection step corresponding to the above step 13 is the above-mentioned plurality. It is related to the distribution of the defined gaze area of the type in a given three-dimensional space. For example, the overall size of the vehicle space of different models can be different, and the positions of the same type of area, such as a glove box, in different vehicle spaces can be different, so in different three-dimensional spaces. The division of multiple types of defined gaze areas can also be different, for example, the number and types of defined gaze areas can be different. Therefore, it is necessary to train different gaze area classifiers for different 3D spaces, for example, different gaze area classifiers for M model vehicles and N model vehicles with different spatial distributions. ..

Therefore, the same method can be used to perform line-of-sight detection for vehicles of different models, requiring only retraining of the gaze area classifier when changing vehicle models. Training a gaze area classifier is relatively easy, requires less data, and is faster than training an entire convolutional neural network in an end-to-end fashion, so different vehicles The time cost and technical difficulty in transferring the above gaze area detection method between models can be significantly reduced.

In another embodiment of the invention, the gaze area detection method may further comprise obtaining a gaze area classifier that has been trained for the predetermined three-dimensional space prior to step 11. In the present invention, the following method 1 or method 2 can be adopted to obtain a gaze area classifier for which training for the predetermined three-dimensional space has been completed.

In method 1, when it is necessary to perform gaze area detection, a gaze area classifier for a predetermined three-dimensional space is trained in real time.

As shown in FIG. 2, training the gaze area classifier for a predetermined three-dimensional space in real time causes the gaze start point information and the gaze direction information of at least one gaze feature sample to be the gaze area classifier waiting for training. Based on the deviation between the step 101 of inputting to obtain the gaze area type prediction information corresponding to the line-of-sight feature sample and the gaze area type prediction information and the gaze area type labeling information corresponding to the line-of-sight feature sample. , And step 102 of training the gaze area classifier by performing parameter adjustments on the gaze area classifier.

For example, the above-mentioned predetermined three-dimensional space can be the space of a vehicle of a certain model. First, the fixed position of the camera for collecting the face image is determined. For example, the camera is fixed in the position of the center console to collect facial images of the driver in the driving area. Subsequent facial images required for the classifier training and detection stages are all collected using the above camera in the fixed position.

At the same time, gaze area division is performed for different parts of the vehicle above, and multiple types are defined in the vehicle space above, primarily based on the areas that the driver's eyes need to pay attention to during the vehicle driving process. The gaze area is divided, and the corresponding type information is set for each of a plurality of defined gaze areas.

In one embodiment of the present invention, the plurality of types of defined gaze areas obtained by dividing the vehicle space are the left windshield area, the right windshield area, the instrument panel area, the interior mirror area, the center console area, and the like. It may include at least two of the left rear-view mirror area, the right rear-view mirror area, the shading plate area, the shift lever area, the area below the steering wheel, the co-pilot area, and the glove box area in front of the co-pilot.

FIG. 3 is a schematic diagram of a plurality of types of defined gaze areas according to an exemplary embodiment of the present invention. Multiple types such as left windshield, right windshield, instrument panel, interior mirror, center console, left rearview mirror, right rearview mirror, sun visor, shift lever, mobile phone for one given model of vehicle The defined gaze area of can be determined. Corresponding type information can be set in advance for each of a plurality of types of defined gaze areas, and for example, a numerical value can be used to indicate a type value. The correspondence between the plurality of types of defined gaze areas described above and the predetermined type values may be as shown in Table 1.

It is necessary to explain that the above type information can also be indicated by predetermined English characters such as A, B, C ... J.

Then, a facial image sample is collected to obtain a training sample set. The training sample set may include a plurality of line-of-sight feature samples, wherein each line-of-sight feature sample contains line-of-sight start point information, line-of-sight direction information, and labeling information of the gaze area type corresponding to the line-of-sight feature sample. The type of gaze area included and labeled belongs to one of a plurality of defined gaze areas obtained by dividing the predetermined three-dimensional space. Here, how to determine the line-of-sight start point information and the line-of-sight direction information of a person based on a facial image will be described in detail on the rear surface.

Subsequently, using the above training sample set, the classifier for the above-mentioned predetermined three-dimensional space is trained by repeatedly executing the following steps, where the step is the above-mentioned training sample set. The line-of-sight start point information and the line-of-sight direction information of one of the line-of-sight feature samples are input to the gaze area classifier waiting for training to obtain the prediction information of the gaze area type corresponding to the line-of-sight feature sample, and the line of sight. Training the gaze area classifier by performing parameter adjustments to the gaze area classifier based on the deviation between the prediction information for the gaze area type and the labeling information for the gaze area type in the feature sample. And further include.

In one exemplary embodiment, step 102 above obtains a loss function value based on the difference between the predicted value of the gaze area type of one gaze area type and the labeling value of the gaze area type of one gaze feature sample. When the loss function value satisfies the predetermined training end condition, the training is ended, the classifier currently in the training stage is determined as the trained classifier, and the loss function value is the above-mentioned predetermined training. Failure to meet the termination condition may include performing parameter adjustments to the gaze area classifier based on the loss function value.

In an embodiment of the invention, the loss function is a mathematical representation for measuring the degree of misclassification of a classifier model with respect to a training sample during the training process. The loss function value can be obtained based on the entire training sample set, and the larger the above loss function value, the higher the misclassification rate of the classifier at the current training stage, and conversely, the above loss function value. The smaller the value, the smaller the misclassification rate of the classifier at the current training stage.

The above-mentioned predetermined training end condition is a condition for ending the training of the gaze area classifier. In one embodiment, the above-mentioned predetermined training end condition may be that the loss function value of the predetermined loss function is smaller than the predetermined threshold value. Ideally, the predetermined training end condition described above is that the loss function value is equal to zero. This indicates that all gaze area types currently predicted by the classifier are accurate. In actual operation, the above-mentioned predetermined threshold value may be a predetermined one empirical value in consideration of the problems of training efficiency and training cost of the gaze area classifier.

As in the above example, if the current loss function value is greater than or equal to the above predetermined threshold, it means that the accuracy rate of the prediction result of the classifier at the current training stage is not as expected. After adjusting the related parameters of the gaze area classifier using the above loss function value until is satisfied, step 101 and step 102 are repeatedly executed using the gaze area classifier after updating the parameters. Then, it is possible to obtain a gaze area classifier in which the training for the above-mentioned predetermined three-dimensional space is completed.

In an embodiment of the invention, the computer system employs algorithms such as support vector machines, naive bayes, decision trees, random forests, and K-means to train the gaze area classifier described above. Can be done.

In an embodiment of the invention, it is necessary to redefine the training sample set and train the corresponding gaze area classifier in response to changes in a given three-dimensional space. Time cost and technology when transferring the above gaze area detection method between different 3D spaces (eg, different vehicle model spaces) because classifier training does not require large amounts of data and training speeds are faster. Difficulty can be greatly reduced.

In the method 2, when it is necessary to execute the gaze area detection, the gaze area classifier for the above-mentioned predetermined three-dimensional space is directly acquired from the predetermined storage resource.

In one embodiment of the invention, the computer system associates the gaze area classifier, which has been trained for each type of predetermined three-dimensional space, with the spatial identifier of the predetermined three-dimensional space, such as a cloud server. Stores in the specified storage resource to form a given gaze area classifier set. In the intelligent driving application scenario described above, the predetermined gaze area classifier set may include a correspondence between a plurality of vehicle models and the gaze area classifier, as shown in Table 2.

If the computer system of a new vehicle of one known model (eg, model is M01) is not equipped with a gaze area classifier program, the vehicle will have its own model (eg, M01) before performing gaze area detection. ), The corresponding target gaze area classifier program (for example, the computer program corresponding to the above-mentioned first classifier) is automatically downloaded from the cloud server, so that the gaze area detection can be realized quickly.

In the embodiment of the present invention, the line-of-sight detection result obtained by step 12 includes at least the line-of-sight start point information and the line-of-sight direction information of the person in the face image, and the head appearance information of the person in the face image. Can be further included.

According to the embodiment of the present invention, as shown in FIG. 4, by executing steps 1211-1212, the line-of-sight start point information of the person in the face image can be determined.

In step 1211, the position of the eyes in the face image is detected.

In an embodiment of the invention, the eye position is the position of the eye in the facial image in the actual camera coordinate system. The above-mentioned actual camera coordinate system is a spatial Cartesian coordinate system determined by a computer system based on the above-mentioned camera. The camera is a camera that captures the face image in the predetermined three-dimensional space, and can be marked as the camera C0.

The Z-axis of the actual camera coordinate system is the optical axis of the camera, and the optical center of the camera lens is the origin of the predetermined actual camera coordinate system. The X-axis, which is the horizontal axis of the actual camera coordinate system, and the Y-axis, which is the vertical axis, are parallel to the lens plane of the camera.

In the embodiment of the present invention, the computer system can detect the position of the eyes in the facial image by adopting any of the following methods. That is, in the first method, at least two cameras are used to simultaneously collect face images of at least two frames for one target object such as the above driver, and the camera calibration method is used to collect the above-mentioned face. Acquiring the position of the eyes in the image, where at least the two cameras described above include a camera that collects face images waiting to be detected. In the second method, the head shape information of the person in the face image is detected, and the position of the eyes in the face image is detected based on the head shape information.

In one embodiment of the present invention, the computer system utilizes a head shape estimation method in related techniques such as a flexible model method and a geometric method based on a face image taken by one camera. It is possible to determine the head shape information of the driver and acquire the 3D position in the predetermined actual camera coordinate system of the target eye based on the head shape information, and here, the above-mentioned predetermined actual camera. The coordinate system is a camera coordinate system determined based on the above camera C0.

By adopting the second method of determining the position of the eyes described above, the 3D position of the eyes can be determined using the face image collected by a single camera, that is, a monocular camera, and the gaze area can be detected. The hardware configuration cost for this can be saved.

In step 1212, the line-of-sight start point information of the person in the face image is determined based on the position of the eyes.

In the present invention, the eye positions detected from the face image in step 1211 may include the position of one eye of the target object such as the driver in the face image, and the positions of both eyes (that is, the driver's left eye and right eye). (Position of) may be included.

Accordingly, in the embodiment of the present invention, the following method 1 or method 2 can be adopted to determine the line-of-sight start point information of the person in the above facial image.

In the method 1, the line-of-sight start point information of the person in the above facial image is determined based on the position of one eye. In one embodiment, when the eye positions determined in step 1211 include the positions of both eyes, the line-of-sight start point information of the person in the above facial image is determined based on the position of any one eye in the eye positions. can do. In another embodiment, when the eye position determined in step 1211 includes the position of one eye, the line-of-sight start point information of the person in the above facial image can be determined based on the position of the one eye.

In the method 2, when the position of the eye determined in step 1211 includes the position of both eyes, the intermediate position of the eyes is determined as the line-of-sight start point information, and here, the intermediate position of both eyes is the position of both eyes. It can be the midpoint position of the 3D coordinate connection line, or another position on the 3D coordinate connection line of both eyes.

In the embodiment of the present invention, determining the line-of-sight start point information of a person in a face image by adopting the above method 2 is inaccurate in the line-of-sight start point information due to one-eyed detection error as compared with the above method 1. It is useful to eliminate the problem and improve the accuracy of the line-of-sight detection result.

According to the embodiment of the present invention, as shown in FIG. 5, the line-of-sight direction information of the person in the face image can be detected by executing steps 1221 to 1222.

In step 1221, the head appearance information of the person in the face image is detected.

As mentioned above, the computer system utilizes the head shape estimation method in the related technology such as flexible model method, geometric method, etc. based on the face image taken by one camera, and the above driver. The head shape information can be confirmed.

The above-mentioned flexible model method includes an active shape model (Active Shape Model, ASM), an active appearance model (Active Appealance Model, AAM), and an elastic map matching model (Elastic Graph) for the facial composition of the head image in the image plane. Matching flexible models such as Matching, EGM) to obtain the final result of head shape estimation by feature comparison or model parameters.

Geometric method refers to estimating the shape of the head using the shape of the head and accurate morphological information of local facial features such as the relevant positions of the eyes, nose, and mouth. ..

According to the embodiment of the present invention, the head shape of a person in an image can be estimated based on a single frame image collected by a monocular camera.

According to the embodiment of the present invention, as shown in FIG. 6, by executing steps 1201 to 1202, it is possible to detect the head shape information of the person in the facial image (step 1221).

In step 1201, a plurality of face key points in the face image are detected.

In one embodiment of the present invention, face keypoint detection may be performed using an edge detection algorithm such as a Robert algorithm or a Sobel algorithm, or a related model such as an active contour model (for example, a Snake model) may be used. Face keypoint detection may be performed.

In another embodiment of the present invention, face keypoint detection can be performed using a neural network for performing face keypoint detection. It should be noted that a third party application (eg, Dlib toolkit, etc.) can be used to perform face keypoint detection.

Using the above method, the facial key points include the position coordinates of the face key points such as the left eye corner, the right eye corner, the tip of the nose, the left mouth corner, the right mouth corner, and the lower jaw in a predetermined quantity (for example, 160 pieces). The position can be detected. It can be understood that the number of face keypoint position coordinates obtained may differ depending on the face keypoint detection method. For example, the Dlib toolkit can be used to detect 68 facial keypoint positions.

In step 1202, the detected face key points and a predetermined average face model are used to determine the head appearance information of the person in the face image.

Returning to FIG. 5, in step 1222, the line-of-sight direction information of the person in the face image is detected based on the head figure information.

In the embodiment of the present invention, it is possible to detect the line-of-sight direction information of a person in the above facial image by using an already trained neural network based on the head posture information.

Referring to FIG. 7, the step 1222 may include steps 12221-12223.

In step 12221, the face image is normalized based on the head shape information to obtain a normalized face image.

In the actual operation, in the case of face images collected at different time points by the camera C0, the position in the entire face area image is randomly changed, and the head shape of the person in the image is also randomly changed. When using the face image collected directly by the camera as a sample image when training the above neural network, the randomness of the head shape and face area image position increases the training difficulty and training time of the neural network. There is no doubt about it.

According to the embodiment of the present invention, when training the neural network for detecting the above-mentioned line-of-sight direction, in order to reduce the training difficulty, first, it is normal for each sample image data in the training sample set. By executing the normalization process, the sample image data after the normalization process is used after the sample image data after the normalization process is made equal to the image data taken by the virtual camera facing the head. And train the neural network.

In response to this, at the application stage of the neural network, in order to ensure the accuracy of detection of the line-of-sight direction information, the face image is first normalized, and then the normalization in the corresponding virtual camera coordinate system is performed. It is necessary to obtain the converted facial image and input it into the above neural network to detect the line-of-sight direction information.

Referring to FIG. 8A, the above step 12221 may include steps 12-1-12-3.

In step 12-1, the head coordinate system of the person in the face image is determined based on the head shape information. For example, the X-axis of the head coordinate system is parallel to the connecting line of the coordinates of the left eye and the right eye, and the Y-axis of the head coordinate system is the plane of the face and perpendicular to the X-axis. The Z-axis of the coordinate system is perpendicular to the plane of the face, and the starting point of the line of sight is the origin of the head coordinate system.

In the embodiment of the present invention, the computer system detecting the head shape information of the target target based on the above facial image is equivalent to the computer system predicting the three-dimensional head model of the target target. The three-dimensional head model can show the appearance information of the target head with respect to the camera C0 when the camera C0 collects the above facial image. Based on this, the computer system can determine the head coordinate system of the target target based on the head shape information.

The head coordinate system can indicate a spatial Cartesian coordinate system. The X-axis of the head coordinate system is parallel to the connecting line of the 3D position coordinates of both eyes in the three-dimensional head model. The midpoint of the connecting line of the 3D position coordinates of both eyes, that is, the starting point of the line of sight can be determined as the origin of the head coordinate system. The Y-axis of the head coordinate system is perpendicular to the X-axis on the face. The Z axis of the head coordinate system is perpendicular to the face of the face.

In step 12-2, a virtual camera coordinate system is obtained by performing rotation and translation with respect to the actual camera coordinate system corresponding to the face image based on the head coordinate system. For example, the Z axis of the virtual camera coordinate system points to the origin of the head coordinate system, and the X axis of the virtual camera coordinate system and the X axis of the head coordinate system are on the same plane, and the virtual camera. The origin of the coordinate system and the origin of the head coordinate system are separated by a predetermined distance in the Z-axis direction of the virtual camera coordinate system.

In the embodiment of the present invention, the computer system determines the head coordinate system of the target target, and then performs a rotation or translation operation with respect to the above camera with reference to the above head coordinate system. One virtual camera can be determined, and a virtual camera coordinate system corresponding to the virtual camera can be constructed based on the position of the virtual camera in the head coordinate system. The method for constructing the virtual camera coordinate system is the same as the method for constructing the predetermined actual camera coordinate system, that is, the Z axis of the virtual camera coordinate system is the optical axis of the virtual camera, and the virtual camera is described above. The X-axis and Y-axis of the camera coordinate system are parallel to the lens plane of the virtual camera, and the optical center of the virtual camera lens is the origin of the virtual camera coordinate system.

The positional relationship between the virtual camera coordinate system and the head coordinate system satisfies the following three conditions.

Condition 1 is that the Z axis of the virtual camera coordinate system points to the origin of the head coordinate system.

Condition 2 is that the X-axis of the virtual camera coordinate system is located on the same plane as the X-axis of the head coordinate system, and here, the X-axis of the virtual camera coordinate system and the X-axis of the head coordinate system. The relative positional relationship with and, including, but not limited to, parallel relationships.

The condition 3 is that the origin of the virtual camera coordinate system is separated from the origin of the head coordinate system by a predetermined distance in the Z-axis direction of the virtual camera coordinate system.

The above process is equivalent to performing the following operations on the camera C0 to determine one virtual camera, that is, rotating the camera C0 and its Z-axis in the eye image. At the same time as pointing to the starting point of the person's 3D line of sight, the X-axis of the camera C0 should be on the same plane as the X-axis of the above-mentioned head coordinate system, and the rotated camera C0 should be parallel to the Z-axis. Move so that the distance between the center of light of the lens and the origin of the above-mentioned head coordinate system is a predetermined length.

So far, computer systems have actually based on the positional relationship between the actual camera coordinate system and the head coordinate system, and the positional relationship between the virtual camera coordinate system and the head coordinate system described above. It is possible to determine the position conversion relationship between the camera coordinate system of the above and the virtual camera coordinate system described above.

It should be understood that in the present invention, different facial images may correspond to different virtual camera coordinate systems because the virtual camera coordinate system is related to the head shape of the person in the facial image.

In step 12-3, the face image is normalized based on the position conversion relationship between the actual camera coordinate system and the virtual camera coordinate system, and the normalized face image is obtained. obtain.

In the embodiment of the present invention, the computer system utilizes the position conversion relationship between the actual camera coordinate system and the virtual camera coordinate system to perform rotation, affine, zoom conversion, and the like with respect to the face image. Processing can be performed to obtain a normalized face image in the virtual camera coordinate system described above.

FIG. 8B is a schematic diagram showing that the acquired face image according to an exemplary embodiment is subjected to normalization processing, where the image P0 is collected by the actual vehicle-mounted camera C0 for the driver. The image P1 shows a normalized face image in the virtual camera coordinate system obtained after the above normalization process, that is, collected by one virtual camera C1 facing the driver's head. Corresponds to the driver's face image.

Returning to FIG. 7, in step 12222, the line-of-sight direction detection is executed based on the normalized face image to obtain the first detected line-of-sight direction. For example, the first detected line-of-sight direction is three-dimensional line-of-sight direction information in the virtual camera coordinate system, and may be a three-dimensional direction vector.

In an embodiment of the present invention, a person in the normalized facial image described above is input to a neural network for detecting a trained line-of-sight direction by inputting a normalized facial image through the normalized process. 3D line-of-sight direction information can be detected. The neural network for detecting the line-of-sight direction may include a deep neural network (DNP) such as a convolutional neural network (CNN).

In step 12223, the coordinate inverse conversion process is performed on the first detected line-of-sight direction to obtain the line-of-sight direction information of the person in the face image.

In the subsequent gaze area detection stage, it is necessary to input the gaze feature vector in the actual camera coordinate system to the gaze area classifier. Therefore, in the present invention, after the computer system detects the above-mentioned first detection line-of-sight direction which is the line-of-sight direction information in the virtual camera coordinate system, the above-mentioned actual operation is performed from the virtual camera coordinate system with respect to the upper first detection line-of-sight direction. It is necessary to execute the coordinate inverse conversion process up to the camera coordinate system to obtain the line-of-sight direction information in the above actual camera coordinate system.

Returning to FIG. 1, the above step 12 corresponds to the process of determining the line-of-sight feature vector of the person in the face image, and the line-of-sight feature vector obtains the line-of-sight start point information and the line-of-sight direction information of the person in the face image. include.

For example, in the actual application of intelligent driving, the process of extracting the line-of-sight feature vector for the above facial image is not changed by changing the vehicle model. The artificial neural network used at this stage (neural network for detecting face key points, neural network for detecting line-of-sight direction, etc.) can be applied to different vehicle models and has good mobility.

As described above, according to one embodiment of the present invention, in step 13, training of the line-of-sight start point information and the line-of-sight direction information of the person in the face image determined in step 12 for a predetermined three-dimensional space has already been completed. It is possible to detect the type of the target gaze area corresponding to the face image by inputting to the gaze area classifier.

In the embodiment of the present invention, the above step 13 may include determining the target gaze area information based on the type of the target gaze area and outputting the target gaze area information.

For example, the classifier can output the type of target gaze area as shown in FIG. 9A, or can directly output the name of the target gaze area as shown in FIG. 9B.

In another embodiment of the invention, the gaze area detection method may further comprise training a neural network to detect the gaze direction prior to step 11. The step corresponds to the training process of the three-dimensional line-of-sight direction estimation model. It is necessary to explain that the step is the process of training the gaze area classifier in real time shown in FIG. 2 and that it can be performed on different computer systems.

FIG. 10 is a flowchart of a method for training a neural network for detecting a three-dimensional line-of-sight direction according to an exemplary embodiment of the present invention. The method may include steps 1001-1005.

In step 1001, the original sample set containing at least one face sample is determined, where each face sample contains a face image sample and gaze direction labeling information.

In the embodiment of the present invention, the above neural network can be trained by adopting a supervised learning method. Correspondingly, each sample in the sample set for training the above neural network is an input information for prediction, that is, a face image sample, and a true value corresponding to the input information, that is, an actual camera. It may include the actually detected line-of-sight information in the coordinate system. In the embodiment of the present invention, the above-mentioned actually detected line-of-sight direction information is also referred to as line-of-sight direction labeling information.

In step 1002, the head figure information corresponding to each of the face image samples is determined based on the face key points and the average face model.

In step 1003, based on the head shape information and the actual camera coordinate system, the virtual line-of-sight in the virtual coordinate system of the normalized face image sample corresponding to each face image sample and the line-of-sight labeling information. Confirm the direction labeling information.

The implementation process of the above steps 1002 and 1003 is the same as the above steps 1202 and steps 12-1 to 12-3, respectively, and will not be described repeatedly here. At the same time, the computer system converts the above-mentioned line-of-sight direction labeling information into virtual line-of-sight labeling information based on the position conversion relationship from the actual camera coordinate system to the virtual camera coordinate system.

So far, we have obtained a sample set in the virtual camera coordinate system. Subsequently, using the sample set, the following steps were repeatedly trained until the training requirements of the neural network for detecting the three-dimensional line-of-sight direction were satisfied, and these steps were each normalized. The deviation between the step 1004 of inputting the face image sample into the three-dimensional line-of-sight detection neural network waiting for training to obtain the three-dimensional line-of-sight direction prediction information and the three-dimensional line-of-sight direction prediction information and the virtual line-of-sight direction labeling information. Includes step 1005, which performs parameter adjustments to the neural network to obtain a neural network for detecting line-of-sight information.

In the embodiment of the present invention, by adopting the normalized face image after the normalization processing in the virtual camera coordinate system as the training sample data, the difficulty of training the neural network due to the change in the head shape is reduced, and the line-of-sight direction It is possible to improve the training efficiency of the neural network for detecting.

As an example, after recognizing the gaze area of the driver, further operations can be performed based on the gaze area. For example, the attention monitoring result of the person corresponding to the face image can be determined based on the gaze area type detection result. For example, the gaze area type detection result may be the type of the gaze area within a predetermined time zone. Illustratively, the gaze area type detection result may be "at a predetermined time zone, the gaze area of the driver is always area 2." If the area 2 is the right windshield, it means that the driver is devoting himself to driving. If the area 2 is the glove box area in front of the co-pilot, it means that the driver is likely to be distracted and unable to concentrate.

After detecting the attention monitoring result, the attention monitoring result can be output, and for example, "driving is well devoted" can be displayed in a certain display area in the vehicle. Alternatively, distraction prompt information can be output based on the attention monitoring result, and the distraction prompt information can be displayed promptly on the display screen, or a voice prompt or the like can be used to "in order to ensure driving safety. Focus your attention, "prompts the driver. As a matter of course, when the information is specifically output, at least one of the attention monitoring result and the distraction prompt information can be output.

By confirming human attention monitoring results based on the detection of the gaze area type and outputting distraction prompt information, it is an important help for the driver's attention monitoring, and the driver is focusing his attention. It can effectively detect situations that are not present, quickly remind them, reduce the risk of accidents, and ensure driving safety.

In the description of the above example, an example of monitoring the driver's attention in an intelligent driving application scenario will be described. Besides this, gaze area detection has many other uses.

For example, interactive control of the vehicle and machine based on gaze area detection can be performed. Some electronic devices such as multimedia players can be installed in the vehicle. By detecting the gaze area of a person in the vehicle, the multimedia player can be automatically controlled to activate the playback function based on the detection result of the gaze area.

Illustratively, a camera placed in the vehicle is used to capture a facial image of a person (driver or passenger, etc.) in the vehicle, and a pre-trained neural network is used to detect the gaze area type. Is detected. For example, the detection result may be that in time zone T, the gaze area 1 of a person in the vehicle is always the area where the "gaze activation" option on a multimedia player in the vehicle is located. .. Based on the above detection result, it can be confirmed that the person in the vehicle is trying to activate the multimedia player, so that the corresponding control command is output so that the multimedia player starts executing playback. Can be controlled to.

In addition to vehicle-related applications, it may further include scenarios for multiple types of applications such as game control, smart home device control, and ad push. To give an example of smart home control, it is possible to collect a face image of a controller and detect a gaze area type detection result via a pre-trained neural network. For example, the detection result may be that in the time zone T, the gaze area 1 of the controller is always the area where the "gaze activation" option on the smart air conditioner is located. Based on the above detection result, it can be determined that the controller is about to start the smart air conditioner, so that the corresponding control command can be output to control the air conditioner to be started.

For convenience of explanation, examples of each of the above-mentioned methods are described in a series of operation combinations. Those skilled in the art should appreciate that the invention is not limited to the sequence of operations described. According to the present invention, some steps may adopt other sequences or be performed simultaneously.

INDUSTRIAL APPLICABILITY The present invention can further provide examples of devices and electronic devices corresponding to the embodiments of the above-mentioned method.

FIG. 11 is a block diagram of the gaze area detection device 1100 according to an exemplary embodiment of the present invention. The gaze area detection device 1100 may include an image acquisition module 21, a gaze detection module 22, and a gaze area detection module 23.

The image acquisition module 21 acquires a face image collected in a predetermined three-dimensional space. The line-of-sight detection module 22 executes line-of-sight detection based on the face image and obtains a line-of-sight detection result. In one embodiment of the present invention, the line-of-sight detection result may include line-of-sight start point information and line-of-sight direction information of a person in the face image. The gaze area detection module 23 detects the type of the target gaze area corresponding to the face image based on the gaze detection result by using the gaze area classifier trained in advance for the predetermined three-dimensional space. do. The target gaze area belongs to one of a plurality of defined gaze areas obtained by preliminarily dividing the predetermined three-dimensional space.

Referring to FIG. 12, the line-of-sight detection module 22 of the gaze area detection device according to the exemplary embodiment of the present invention includes an eye position detection submodule 221 for detecting the position of an eye in the face image and the eye. When the position of the eyes includes the positions of both eyes, a first start point information determination submodule 222 for determining the intermediate position of the eyes as the line-of-sight start point information may be provided.

Referring to FIG. 13, another line-of-sight detection module 22 of the gaze area detection device according to the exemplary embodiment of the present invention includes an eye position detection submodule 221 for detecting the position of the eyes in the face image. , If the position of the eyes includes the positions of both eyes, the position of any one eye in the eyes is determined as the line-of-sight start point information, or if the position of the eyes includes the position of one eye, the one eye. A second start point information determination submodule 223 for determining the position of the line of sight as the line-of-sight start point information may be provided.

Referring to FIG. 14, the eye position detection submodule 221 in FIGS. 12 and 13 according to an exemplary embodiment of the present invention is a shape detection unit for detecting head shape information of a person in the face image. 2211 and a position determination unit 2212 for determining the position of the eyes in the face image based on the head shape information may be provided.

Referring to FIG. 15, another line-of-sight detection module 22 of the gaze area detection device according to the exemplary embodiment of the present invention is a shape detection submodule for detecting head shape information of a person in the face image. 22-1 and a direction detection submodule 22-2 for detecting line-of-sight direction information of a person in the face image based on the head shape information may be provided.

Referring to FIG. 16, the figure detection submodule 22-1 in FIG. 15 according to an exemplary embodiment of the present invention is a key point detection unit 22- for detecting a plurality of face key points in the face image. 11 and a figure determining unit 22-12 for determining the head figure information of the person in the face image based on the face key points and a predetermined average face model may be provided.

Referring to FIG. 17, the direction detection submodule 22-2 in FIG. 15 according to an exemplary embodiment of the present invention performs a normalization process on the face image based on the head shape information. An image processing unit 22-21 for obtaining a normalized face image, and a first-direction detection unit for executing line-of-sight direction detection based on the normalized face image to obtain a first detected line-of-sight direction. 22-22 may be provided with a direction determination unit 22-23 for performing a coordinate inverse conversion process with respect to the first detected line-of-sight direction to obtain line-of-sight direction information of a person in the face image.

Referring to FIG. 18, the image processing unit 22-21 in FIG. 17 according to an exemplary embodiment of the present invention determines the head coordinate system of a person in the face image based on the head figure information. To obtain a virtual camera coordinate system by rotating and translating the head coordinate determination subunit 22-211 for the purpose and the actual camera coordinate system corresponding to the face image based on the head coordinate system. Based on the position conversion relationship between the coordinate conversion subunit 22-212 of the above and the actual camera coordinate system and the virtual camera coordinate system, the face image is normalized by performing the normalization process. An image processing subsystem 22-213 for obtaining a face image may be provided.

In an embodiment of any of the above-mentioned devices of the invention, the gaze area classifier can be pre-trained based on a training sample set for the predetermined three-dimensional space. The training sample set may include a plurality of line-of-sight feature samples, and each line-of-sight feature sample contains line-of-sight start point information, line-of-sight direction information, and labeling information of the gaze area type corresponding to the line-of-sight feature sample, and is labeled. The type of gaze area is one of a plurality of defined gaze areas obtained by dividing the predetermined three-dimensional space.

FIG. 19 is a block diagram of another gaze area detection device 1900 according to an exemplary embodiment of the present invention. Compared to the gaze area detection device 1100 shown in FIG. 11, the gaze area detection device 1900 may further include a classifier training module 20.

The classifier training module 20 inputs the line-of-sight start point information and the line-of-sight direction information of at least one line-of-sight feature sample into the gaze area classifier waiting for training, and gaze area type prediction information corresponding to the line-of-sight feature sample. Parameter adjustment for the gaze area classifier based on the deviation between the gaze area type prediction information and the gaze area type labeling information corresponding to the gaze feature sample. May further include a parameter adjustment submodule 202 for training the gaze area classifier.

FIG. 20 is a block diagram of another gaze area detection device 2000 according to an exemplary embodiment of the present invention. Compared to the gaze area detection device 1100 shown in FIG. 11, the gaze area detection device 2000 may further include a classifier acquisition module 203.

The classifier acquisition module 203 can acquire a gaze area classifier corresponding to the space identifier from a predetermined gaze area classifier set based on the spatial identifier of the predetermined three-dimensional space. The predetermined gaze area classifier set may include gaze area classifiers corresponding to different spatial identifiers in different three-dimensional spaces.

In an embodiment of any of the above-mentioned devices of the present invention, the predetermined three-dimensional space may include a vehicle space. Accordingly, the facial image can be determined based on the image collected for the driving area in the vehicle space. The plurality of types of defined gaze areas obtained by dividing the predetermined three-dimensional space are the left windshield area, the right windshield area, the instrument panel area, the interior mirror area, the center console area, and the left rear-view mirror area. , Right rearview mirror area, shading plate area, shift lever area, steering wheel lower area, co-pilot area, glove box area in front of co-pilot.

FIG. 21 is a block diagram of another gaze area detection device 2100 according to an exemplary embodiment of the present invention. Compared with the gaze area detection device 1100 shown in FIG. 11, the gaze area detection device 2100 has the attention monitoring result of the person corresponding to the face image based on the gaze area type detection result obtained by the gaze area detection module 23. The attention monitoring module 24 for confirming the above, and the monitoring result output module 25 for outputting the attention monitoring result and / or the distraction prompt information based on the attention monitoring result are further provided. obtain.

FIG. 22 is a block diagram of another gaze area detection device 2200 according to an exemplary embodiment of the present invention. Compared with the gaze area detection device 1100 shown in FIG. 11, the gaze area detection device 2200 has a control command confirmation module 26 for determining a control command corresponding to the gaze area type detection result obtained by the gaze area detection module 23. Further, an operation control module 27 for controlling the electronic device to perform the operation corresponding to the control command may be provided.

In the case of the embodiment of the apparatus, since it basically corresponds to the embodiment of the method, the related part may refer to the description of the part of the embodiment of the method. The embodiment of the above device is merely schematic. Here, the unit described as a separation part may or may not be physically separated, and the part displayed as a unit may or may not be a physical unit. be. It can be located in one location or distributed across multiple network units. One of ordinary skill in the art can realize an embodiment of the present invention by selecting some or all of the modules thereof according to actual needs without any creative work.

The present invention can further provide an electronic device corresponding to the above-mentioned gaze area detection method. FIG. 23 is a block diagram of an electronic device 2300 according to an exemplary embodiment of the present invention. For example, the electronic device 2300 may include a processor, an internal bus, a network interface, an internal memory, and a non-volatile memory. The processor can logically form a gaze area detection device for realizing the above-mentioned gaze area detection method by reading a corresponding computer program from the non-volatile memory into the internal memory and operating the processor.

Those skilled in the art should understand that the invention can be provided as a method, device, system, or computer program product. Accordingly, the present invention may adopt embodiments of complete hardware, complete software, or a combination of software and hardware.

The present invention can further provide a computer-readable recording medium, in which the computer program is stored, and when the computer program is executed by the processor, the processor can be used in any of the above-mentioned methods. To realize the gaze area detection method of the embodiment.

Examples of the subject matter and functional operation in the present invention are digital electronic circuits, tangible computer software or firmware, computer hardware including the configurations and structural equivalents thereof disclosed in the present invention, or one or more thereof. It can be realized by combination. The embodiments of the subject in the present invention can be realized as one or more computer programs, i.e., encoded on a tangible non-temporary program carrier and executed by a data processing apparatus. , Can be implemented as one or more modules in a computer program instruction to control the operation of the data processing device. Alternatively or additionally, the program instruction can be encoded on a manually generated propagation signal, for example, on a machine-generated electrical, optical, or electromagnetic signal. can. The signal is generated to encode the information and transmit it to the appropriate receiver device for execution by the data processing device. The computer storage medium can be a machine-readable storage device, a machine-readable storage board, a random or serial access memory device, or a combination thereof.

The processing and logical flow in the present invention can be performed by one or more programmable computers running one or more computer programs, performing operations based on input data to produce output. By doing so, the corresponding function is executed. The processing and logic flow can be further executed by a dedicated logic circuit such as FPGA (field programmable gate array) or ASIC (dedicated integrated circuit), and the device can also be realized as a dedicated logic circuit. Can be done.

Suitable computers for running computer programs include, for example, general purpose and / or dedicated microprocessors, or any other type of central processing unit. Generally, the central processing unit will receive instructions and data from read-only memory and / or random access memory. The basic components of a computer include a central processing unit for executing or executing instructions, and one or more memory devices for storing instructions and data. In general, a computer further includes or is operable with one or more mass storage devices for storing data, such as magnetic disks, magnetic optical disks, or optical disks. Combined with, receive data, transmit data, or have both. However, computers do not necessarily have such devices. The computer can be embedded in another device, such as a mobile phone, personal digital assistant (PDA), mobile audio or video player, game console, Global Positioning System (GPS) receiver, or general purpose serial bus. It can be embedded in portable storage devices such as (USB) flash drives, and these devices are just a few examples.

Computer-readable media suitable for storing computer program instructions and data include various forms of non-volatile memory, intermediaries, and memory devices, such as semiconductor memory devices (eg, erasable programmable read-only memory (Erasable Programmable)). Read Only Memory (EPROM), electrically erasable programmable read-only memory (Electrically Erasable Read-Only Memory, EEPROM) and flash memory), magnetic disks (eg, internal hard disks or mobile disks), magnetic optical disks, optical disks. Includes read-only memory (Compact Computer Read Only Memory, CD-ROM), digital versatile optical disc (Digital Versaille Disc, DVD), and the like. Processors and memory can be complemented by dedicated logic circuits or incorporated into dedicated logic circuits.

The present invention contains many specific implementation details, which should not be construed as limiting the scope of the invention or the scope of which it seeks to protect, primarily of some embodiments of the invention. Used to describe features. Specific features in a plurality of embodiments of the present invention may also be implemented in combination with a single embodiment. On the other hand, the various features in a single embodiment can be implemented separately in multiple embodiments or in any suitable subcombination. It should be noted that the features play a role in a particular combination as described above and are claimed to be protected in this way from the beginning, but one or more features from the combination claimed to be protected may be excluded from the combination in some cases. And the combinations claimed to be protected can be sub-combined or directed to variants of the sub-combination.

Similarly, the drawings depict operations in a particular order, which either requires them to be performed in a specific order or requires them to be performed in sequence, or all the operations illustrated. It should not be understood that it is a requirement that the expected result be achieved by being carried out. It should be noted that the separation of the various system modules and components in the above embodiments should not be understood to be such separation in all embodiments, and the described program components and systems are: In general, it should be understood that they can be integrated together into a single software product or packaged into multiple software products.

The above are only a few embodiments of the invention and are not used to limit the invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the invention should be included within the scope of the invention.

Claims (34)

It is a gaze area detection method,
Acquiring facial images collected in a predetermined 3D space,
To obtain the line-of-sight detection result by executing the line-of-sight detection based on the face image,
Includes detecting the type of target gaze area corresponding to the face image based on the gaze detection result using a gaze area classifier pre-trained for the predetermined three-dimensional space.
Here, the gaze area detection method, characterized in that the target gaze area belongs to one of a plurality of types of defined gaze areas obtained by preliminarily dividing the predetermined three-dimensional space. The gaze area detection method according to claim 1, wherein the gaze detection result includes gaze start point information and gaze direction information of a person in the face image. To obtain the line-of-sight detection result by executing the line-of-sight detection based on the face image
Detecting the position of the eyes in the facial image and
The gaze area detection method according to claim 2, wherein when the position of the eyes includes the positions of both eyes, the intermediate position of the eyes is determined as the line-of-sight start point information. To obtain the line-of-sight detection result by executing the line-of-sight detection based on the face image
Detecting the position of the eyes in the facial image and
When the position of the eyes includes the positions of both eyes, the position of any one eye in the eyes is determined as the line-of-sight start point information, or when the position of the eyes includes the position of one eye, the position of the one eye. The gaze area detection method according to claim 2, wherein the position is determined as the line-of-sight start point information, and the present invention includes. Detecting the position of the eyes in the facial image is
Detecting the head shape information of a person in the face image and
The gaze area detection method according to claim 3 or 4, wherein the position of the eyes in the facial image is determined based on the head shape information. To obtain the line-of-sight detection result by executing the line-of-sight detection based on the face image
Detecting the head shape information of a person in the face image and
The gaze area detection method according to claim 2, further comprising detecting gaze direction information of a person in the face image based on the head shape information. Detecting the head shape information of a person in the face image is
Detecting multiple face key points in the face image and
The gaze area detection method according to claim 5 or 6, wherein the gaze area detection method according to claim 5 or 6, comprising determining the head appearance information of a person in the face image based on the face key points and a predetermined average face model. .. Detecting the line-of-sight direction information of a person in the face image based on the head figure information is
The face image is normalized based on the head shape information to obtain a normalized face image.
To obtain the first detected line-of-sight direction by executing the line-of-sight direction detection based on the normalized face image,
The gaze area detection according to claim 6 or 7, wherein the first detection line-of-sight direction is subjected to coordinate inverse conversion processing to obtain information on the line-of-sight direction of a person in the face image. Method. It is not possible to obtain a normalized face image by performing normalization processing on the face image based on the head shape information.
Determining the head coordinate system of the person in the face image based on the head figure information,
To obtain a virtual camera coordinate system by rotating and translating the actual camera coordinate system corresponding to the face image based on the head coordinate system.
Includes that the face image is normalized to obtain the normalized face image based on the position conversion relationship between the actual camera coordinate system and the virtual camera coordinate system. The gaze area detection method according to claim 8, wherein the gaze area is detected. The gaze area classifier is pre-trained based on a training sample set for the predetermined three-dimensional space, where the training sample set includes a plurality of gaze feature samples and each gaze feature sample is gaze. The gaze area type including the start point information, the line-of-sight direction information, and the gaze area type labeling information corresponding to the line-of-sight feature sample is the plurality of types obtained by dividing the predetermined three-dimensional space. The gaze area detection method according to any one of claims 1 to 9, wherein the gaze area belongs to one of the defined gaze areas. Before acquiring the facial image collected in the predetermined three-dimensional space,
The line-of-sight start point information and the line-of-sight direction information of at least one line-of-sight feature sample are input to the gaze area classifier waiting for training to obtain gaze area type prediction information corresponding to the line-of-sight feature sample.
Based on the deviation between the gaze area type prediction information and the gaze area type labeling information corresponding to the gaze feature sample, parameter adjustment is performed for the gaze area classifier to obtain the gaze area classifier. The gaze area detection method according to claim 10, further comprising training. Before acquiring the facial image collected in the predetermined three-dimensional space,
Further comprising acquiring a gaze area classifier corresponding to the space identifier from a predetermined gaze area classifier set based on the spatial identifier of the predetermined three-dimensional space.
Here, the gaze area detection method according to claim 10, wherein the predetermined gaze area classifier set includes a gaze area classifier corresponding to each of the spatial identifiers of different three-dimensional spaces. The gaze area detection method according to any one of claims 1 to 12, wherein the predetermined three-dimensional space includes a vehicle space. The facial image is determined based on the image collected for the driving area in the vehicle space.
The plurality of defined gaze areas are left windshield area, right windshield area, instrument panel area, interior mirror area, center console area, left rearview mirror area, right rearview mirror area, shading plate area, shift lever. The gaze area detection method according to claim 13, wherein the gaze area detection method includes at least two types of an area, a lower area of the steering wheel, a sub-pilot area, and a glove box area in front of the sub-pilot. Based on the gaze area type detection result, the attention monitoring result of the person corresponding to the face image is determined, and
One of claims 1 to 14, further comprising outputting the attention monitoring result and / or outputting distraction prompt information based on the attention monitoring result. The gaze area detection method described in the section. Confirming the control command corresponding to the gaze area type detection result and
The gaze area detection method according to any one of claims 1 to 15, further comprising controlling an electronic device to perform an operation corresponding to the control command. It is a gaze area detection device,
An image acquisition module for acquiring face images collected in a predetermined three-dimensional space, and
A line-of-sight detection module for executing line-of-sight detection based on the face image and obtaining a line-of-sight detection result,
A gaze area detection module for detecting the type of the target gaze area corresponding to the face image based on the gaze detection result by using the gaze area classifier trained in advance for the predetermined three-dimensional space. And, with
Here, the gaze area detection device is characterized in that the target gaze area belongs to one of a plurality of types of defined gaze areas obtained by preliminarily dividing the predetermined three-dimensional space. The gaze area detection device according to claim 17, wherein the gaze detection result includes gaze start point information and gaze direction information of a person in the face image. The line-of-sight detection module is
An eye position detection submodule for detecting the position of the eyes in the face image,
18. The invention according to claim 18, wherein when the position of the eyes includes the positions of both eyes, a first start point information determination submodule for determining the intermediate position of the eyes as the line-of-sight start point information is provided. Gaze area detector. The line-of-sight detection module is
An eye position detection submodule for detecting the position of the eyes in the face image,
When the position of the eyes includes the positions of both eyes, the position of any one eye in the eyes is determined as the line-of-sight start point information, or when the position of the eyes includes the position of one eye, the position of the one eye. The gaze area detection device according to claim 18, further comprising a second start point information determination submodule for determining the position as the line-of-sight start point information. The eye position detection submodule
A figure detection unit for detecting the head figure information of a person in the face image, and
The gaze area detection device according to claim 19 or 20, further comprising a position determination unit for determining the position of the eyes in the face image based on the head posture information. The line-of-sight detection module is
A figure detection submodule for detecting the head figure information of a person in the face image,
The gaze area detection device according to claim 18, further comprising a direction detection submodule for detecting gaze direction information of a person in the face image based on the head shape information. The appearance detection submodule is
A key point detection unit for detecting a plurality of face key points in the face image, and
22. The gaze area according to claim 22, comprising: a figure determination unit for determining the head appearance information of a person in the face image based on the face key points and a predetermined average face model. Detection device. The direction detection submodule
An image processing unit for performing normalization processing on the face image based on the head shape information to obtain a normalized face image, and an image processing unit.
A first-direction detection unit for performing line-of-sight detection based on the normalized face image to obtain a first detection line-of-sight direction,
22. 23. Gaze area detector. The image processing unit is
A head coordinate determination subunit for determining the head coordinate system of a person in the face image based on the head figure information,
A coordinate conversion subunit for obtaining a virtual camera coordinate system by rotating and translating the actual camera coordinate system corresponding to the face image based on the head coordinate system.
An image processing subunit for performing normalization processing on the face image based on the position conversion relationship between the actual camera coordinate system and the virtual camera coordinate system to obtain the normalized face image. 24. The gaze area detection device according to claim 24, comprising: The gaze area classifier is pre-trained based on a training sample set for the predetermined three-dimensional space, where the training sample set includes a plurality of gaze feature samples and each gaze feature sample is gaze. The gaze area type including the start point information, the line-of-sight direction information, and the gaze area type labeling information corresponding to the line-of-sight feature sample is the plurality of types obtained by dividing the predetermined three-dimensional space. The gaze area detection device according to any one of claims 17 to 25, which belongs to one of the defined gaze areas. Further equipped with a classifier training module, the classifier training module is
A type prediction sub for inputting the line-of-sight start point information and the line-of-sight direction information of at least one line-of-sight feature sample into a gaze area classifier waiting for training to obtain gaze area type prediction information corresponding to the line-of-sight feature sample. Module and
Based on the deviation between the gaze area type prediction information and the gaze area type labeling information corresponding to the gaze feature sample, parameter adjustment is performed for the gaze area classifier to obtain the gaze area classifier. 26. The gaze area detector according to claim 26, further comprising a parameter adjustment submodule for training. A classifier acquisition module for acquiring a gaze area classifier corresponding to the space identifier from a predetermined gaze area classifier set based on the spatial identifier of the predetermined three-dimensional space is further provided.
26. The gaze area detection device according to claim 26, wherein the predetermined gaze area classifier set includes a gaze area classifier corresponding to a spatial identifier in a different three-dimensional space. The gaze area detection device according to any one of claims 17 to 28, wherein the predetermined three-dimensional space includes a vehicle space. The facial image is determined based on the image collected for the driving area in the vehicle space.
The plurality of defined gaze areas are left windshield area, right windshield area, instrument panel area, interior mirror area, center console area, left rearview mirror area, right rearview mirror area, shading plate area, shift lever. 29. The gaze area detection device according to claim 29, comprising at least two types of an area, a lower area of the steering wheel, a sub-pilot area, and a glove box area in front of the sub-pilot. An attention monitoring module for determining the attention monitoring result of the person corresponding to the face image based on the gaze area type detection result obtained by the gaze area detection module.
17 to 30, further comprising a monitoring result output module for outputting the attention monitoring result and / or outputting distraction prompt information based on the attention monitoring result. The gaze area detection device according to any one of the items. A control command confirmation module for confirming a control command corresponding to the gaze area type detection result obtained by the gaze area detection module, and a control command confirmation module.
The gaze area detection according to any one of claims 17 to 31, further comprising an operation control module for controlling the electronic device to perform an operation corresponding to the control command. Device. A computer-readable recording medium that stores computer programs.
A computer-readable recording medium, wherein when the computer program is executed by a processor, the processor implements the method according to any one of claims 1 to 16. It ’s an electronic device,
Equipped with memory and processor,
Here, the computer program is stored in the memory.
An electronic device according to any one of claims 1 to 16, wherein the method according to any one of claims 1 to 16 is realized when the processor executes the computer program.

Images