CN110555426A - Sight line detection method, device, equipment and storage medium - Google Patents
Sight line detection method, device, equipment and storage medium Download PDFInfo
- Publication number
- CN110555426A CN110555426A CN201910860197.9A CN201910860197A CN110555426A CN 110555426 A CN110555426 A CN 110555426A CN 201910860197 A CN201910860197 A CN 201910860197A CN 110555426 A CN110555426 A CN 110555426A
- Authority
- CN
- China
- Prior art keywords
- image
- key point
- point information
- eye
- face
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/16—Human faces, e.g. facial parts, sketches or expressions
- G06V40/161—Detection; Localisation; Normalisation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/16—Human faces, e.g. facial parts, sketches or expressions
- G06V40/168—Feature extraction; Face representation
- G06V40/171—Local features and components; Facial parts ; Occluding parts, e.g. glasses; Geometrical relationships
Abstract
the embodiment of the invention discloses a sight line detection method, a sight line detection device, sight line detection equipment and a storage medium. The method comprises the following steps: detecting the obtained face image of the user based on a face detection model to obtain face key point information; determining an eye sub-image in the user face image according to the face key point information; determining eyeball key point information according to the eye subimages based on an eyeball detection model; and determining a sight detection result according to the eyeball key point information based on a sight detection model. By the scheme, sight line detection can be realized only through common image acquisition equipment, the requirement on hardware performance is reduced, detection results are accurately obtained based on models with high stability and robustness, the problems of unstable calculated amount and lack of generalization capability caused by artificial feature calculation are solved, and therefore the accuracy and stability of sight line detection are improved.
Description
Technical Field
The embodiment of the invention relates to the technical field of sight line detection, in particular to a sight line detection method, a sight line detection device, sight line detection equipment and a storage medium.
Background
The eye sight detection means that eyeball tracking equipment is used for measuring the movement of eyeballs and pupils of the eyes, and an estimation result of the gazing direction of the eyes is given. The technology is being widely applied to the field of man-machine interaction after decades of development, such as assisting reading of paralyzed patients or disabled persons, assisting channel selection and channel switching of remote control televisions, assisting observation of concentration of students during teaching of teachers and the like.
However, the current visual detection scheme requires additional equipment and the assistance of a sensor to track and record the movement of human eyes, has higher requirements on the placement position of the equipment and personnel adaptation, has certain use limitation, has higher requirements on hardware performance, cannot be applied to a low-cost mobile equipment end, and has limited scenes. In addition, in the human eye sight line detection process, the introduction of 3D modeling and manual features causes unstable calculated amount, poor accuracy and lack of generalization capability, and human eye sight line estimation operation in various scenes cannot be stably completed.
Disclosure of Invention
Embodiments of the present invention provide a method, an apparatus, a device, and a storage medium for line-of-sight detection, so as to solve the problems of poor detection accuracy, instability, and high requirements for hardware performance in line-of-sight detection.
In a first aspect, an embodiment of the present invention provides a line-of-sight detection method, where the method includes:
detecting the obtained face image of the user based on a face detection model to obtain face key point information;
Determining an eye sub-image in the user face image according to the face key point information;
determining eyeball key point information according to the eye subimages based on an eyeball detection model;
and determining a sight detection result according to the eyeball key point information based on a sight detection model.
in a second aspect, an embodiment of the present invention provides a gaze detection apparatus, including:
the face key point detection module is used for detecting the acquired face image of the user based on the face detection model to obtain face key point information;
the eye sub-image determining module is used for determining an eye sub-image in the user face image according to the face key point information;
the eyeball key point information detection module is used for determining eyeball key point information according to the eye subimage based on an eyeball detection model;
And the sight detection module is used for determining a sight detection result according to the eyeball key point information based on a sight detection model.
in a third aspect, an embodiment of the present invention further provides an apparatus, where the apparatus includes:
one or more processors;
A memory for storing one or more programs;
when executed by the one or more processors, cause the one or more processors to implement a gaze detection method as described in any of the embodiments of the invention.
In a fourth aspect, the embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, which when executed by a processor implements the gaze detection method according to any one of the embodiments of the present invention.
The embodiment of the invention detects the obtained face image of the user based on the face detection model to obtain the key point information of the face; determining an eye sub-image in the user face image according to the face key point information; determining eyeball key point information according to the eye subimages based on an eyeball detection model; and determining a sight detection result according to the eyeball key point information based on a sight detection model. By the scheme, sight line detection can be realized only through common image acquisition equipment, the requirement on hardware performance is reduced, detection results are accurately obtained based on models with high stability and robustness, the problems of unstable calculated amount and lack of generalization capability caused by artificial feature calculation are solved, and therefore the accuracy and stability of sight line detection are improved.
Drawings
fig. 1 is a flowchart of a line-of-sight detection method according to a first embodiment of the present invention;
FIG. 2 is a diagram illustrating facial keypoint information according to a first embodiment of the present invention;
Fig. 3 is a flowchart of a line-of-sight detection method according to a second embodiment of the present invention;
FIG. 4 is a schematic view of an eyeball of a human eye according to a second embodiment of the invention;
fig. 5 is a schematic structural diagram of a sight line detection device in a third embodiment of the present invention;
fig. 6 is a schematic structural diagram of an apparatus according to a fourth embodiment of the present invention.
Detailed Description
The present invention will be described in further detail with reference to the accompanying drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the invention and are not limiting of the invention. It should be further noted that, for the convenience of description, only some of the structures related to the present invention are shown in the drawings, not all of the structures.
Example one
fig. 1 is a flowchart of a gaze detection method in a first embodiment of the invention. The sight line detection method provided in this embodiment may be applicable to the case of detecting the sight line of human eyes, and the method may be specifically executed by a sight line detection device, where the device may be implemented in a software and/or hardware manner, and the device may be integrated in an apparatus, where the apparatus is loaded with an image capture device for capturing a facial image of a user, and referring to fig. 1, the method in the embodiment of the present invention specifically includes:
And S110, detecting the obtained face image of the user based on the face detection model to obtain face key point information.
the face detection model is a model obtained by training in advance and is used for detecting the face image of the user. The face key points are the coordinates of the position points of each key point of the face in the face image of the user. For example, five points of a single brow portion of the face of the user are selected as key points, and position point coordinates of the five key points of the single brow portion of the face image of the user can be obtained.
Specifically, when the eye sight of a person is detected, if the artificial features in the facial image of the user are extracted through an algorithm for calculation, and then the direction of realization is mapped through geometry, the introduction of the artificial features causes unstable calculated amount and lack of generalization capability, the accuracy and stability of the eye sight detection are poor, and the detection of the eye sight of the person in various scenes cannot be completed robustly and stably. Therefore, in the embodiment of the present invention, based on the face detection model obtained by pre-training, the face image of the user acquired by the image acquirer is detected, so as to obtain the face key point information, and further obtain the eye sub-image according to the face key point information. By training the face detection model in advance and obtaining the face key point information based on the face detection model, the problems of unstable calculated amount and lack of generalization capability caused by artificial feature introduction are solved, and the accuracy and stability of detection are improved.
And S120, determining the ocular sub-image in the user facial image according to the facial key point information.
Wherein the eye sub-image is an image including eyes of the user and a face of a portion around the eyes. The face key point information comprises key point information of corresponding parts of each organ, and the eye sub-images in the face images of the users can be determined according to the eye key point information, so that sight line detection is further performed according to the eye sub-images. By acquiring the eye sub-image from the face image of the user, the sight line detection is more targeted, and the accuracy of the sight line detection is improved.
and S130, determining eyeball key point information according to the eye subimages based on the eyeball detection model.
The eyeball detection model is a model obtained by pre-training and is used for detecting the eye subimages. The eyeball key point information comprises a pupil position point coordinate and an iris key point position point coordinate.
specifically, the eye gaze detection needs to detect and track according to the positions of the pupil and the iris of the eye, so as to determine the implementation direction of the user according to the positions of the pupil and the iris. Therefore, in the embodiment of the present invention, the eye sub-image is detected by the eye detection model obtained by pre-training, so as to obtain the eye key point information, that is, the coordinates of the pupil position point and the coordinates of the iris key point of the human eye, so as to further determine the sight line. The eyeball key point information is obtained through the eyeball detection model, the problems of large calculation error and inaccurate eyeball key point information caused by conventional algorithm detection are solved, and the accuracy of sight line detection is further improved.
And S140, based on the sight line detection model, determining a sight line detection result according to the eyeball key point information.
The detection realization model is a model obtained through training in advance and is used for detecting the eyeball key point information. The sight line detection result comprises the direction information of the sight line of the human eyes, and the direction information of the sight line of the human eyes can be represented through the posture angle coordinate.
Specifically, the pupil position point coordinates and the iris key point position coordinates in the eyeball key point information are input into the sight line detection model, so that the direction information of the sight line of the human eye is obtained, and the detection and tracking of the sight line of the human eye are realized. The sight line detection result is obtained through the sight line detection model, so that errors caused by direct formula calculation are avoided, and the accuracy of sight line detection is improved.
according to the technical scheme of the embodiment of the invention, the obtained face image of the user is detected based on a face detection model to obtain the key point information of the face; determining an eye sub-image in the user face image according to the face key point information; determining eyeball key point information according to the eye subimages based on an eyeball detection model; and determining a sight detection result according to the eyeball key point information based on a sight detection model. By the scheme, sight line detection can be realized only through common image acquisition equipment, the requirement on hardware performance is reduced, detection results are accurately obtained based on models with high stability and robustness, the problems of unstable calculated amount and lack of generalization capability caused by artificial feature calculation are solved, and therefore the accuracy and stability of sight line detection are improved.
Example two
fig. 3 is a flowchart of a line-of-sight detection method in the second embodiment of the present invention. The present embodiment is optimized based on the above embodiments, and details not described in detail in the present embodiment are described in the above embodiments. Referring to fig. 3, the gaze detection method provided in this embodiment may include:
s210, detecting the obtained face image of the user based on the face detection model to obtain face key point information.
Optionally, the face detection model is constructed as follows: acquiring a sample user face image set, and labeling face key point information on the sample user face image to obtain a labeled user face image set; and taking the marked user face image set as a training data set, and training the convolutional neural network model to obtain the face detection model. Illustratively, a large number of sample user face images are acquired in advance through an image acquisition device to form a sample user face image set, and labels are marked on the sample user face images to form key point information of the sample user face images. And then training the convolutional neural network model by using the labeled user face image set as training data to obtain a face detection model.
S220, segmenting the user face image according to the face key point information to obtain a face region sub-image.
specifically, the face key point information includes key point information of each organ portion, and the user face image can be segmented into sub-images including each organ region according to the key point information of each organ portion. And segmenting the acquired user face image according to the key point information of each organ part in the face key point information to respectively obtain face region sub-images of each organ region of the face. The face area sub-image is obtained by segmenting the face image of the user, so that the sight line detection is more targeted, and the sight line of the human eyes can be accurately detected.
and S230, determining the eye sub-image in the face region sub-image according to the eye key point information in the face key point information.
Specifically, the face key point information includes eye key point information, the eye key point information includes eye socket key point position coordinates and eyeball position point coordinates, and the region image including the eye socket key point position coordinates and the eyeball position point coordinates is used as the eye subimage, so that the eye subimage can be further detected conveniently.
And S240, carrying out gray level processing on the eye sub-image to obtain an eye gray level sub-image.
specifically, the acquired face image of the user may be a color image, and therefore the acquired eye sub-image may be a color image, and the color image increases complexity of line-of-sight detection, and more colors may affect stability of detection.
And S250, acquiring an eye region image at any side in the eye gray level sub-image as a first eye gray level sub-image.
specifically, for the binocular eye region images in the eye gray level sub-image, any one side eye region image is selected as the first eye gray level sub-image.
and S260, carrying out image level mirror image turning processing on the first eye gray level sub-image to obtain a second eye gray level sub-image.
specifically, the first eye gray level sub-image is subjected to image level mirror image turning processing to obtain the second eye gray level sub-image, so that two images with the same characteristics except the initial direction are obtained, detection is only needed once in detection, the sub-image of one side of the eye is only needed to be trained in the model training process, the sub-images of the two sides of the eye are not needed to be detected, and the sight line detection efficiency is improved.
and S270, inputting the first eye gray level sub-image and the second eye gray level sub-image into the eyeball detection model respectively to obtain eyeball key point information output by the eyeball detection model.
Specifically, the two images are input into the eyeball detection model, so that eyeball key point information is obtained, and the implementation direction can be further obtained according to pupil position point information and iris key point position information in the eyeball key point information.
optionally, the eyeball detection model is constructed in the following manner: acquiring a simulated eye subimage and simulated eyeball key point information of a simulated three-dimensional eye model; and training a convolutional neural network by taking the simulated eye subimage and the simulated eyeball key point information as a training data set to obtain the eyeball detection model. Illustratively, a simulated three-dimensional eye model is generated in advance through three-dimensional modeling software, and after a simulated eye sub-image and simulated eyeball key point information of the simulated three-dimensional eye model are obtained, the simulated eye sub-image and the simulated eyeball key point information are used as training data to train a convolutional neural network to obtain an eyeball detection model.
and S280, based on the sight line detection model, determining a sight line detection result according to the eyeball key point information.
Optionally, the gaze detection model is constructed according to the following method: acquiring key point information of a simulated eyeball and simulated sight direction data of a simulated three-dimensional eye model; and training the convolutional neural network by taking the simulated eyeball key point information and the simulated sight direction data as a training data set to obtain a sight detection model, wherein the number of hidden layers of the sight detection model is less than that of the face key point information detection model or the eyeball key point information detection model. Exemplarily, as shown in fig. 4, the information of the key point of the simulated eyeball, i.e. the position coordinate of the pupil or the iris center point a is obtained as (x)0，y0) The coordinate of the eyeball center point b is (x)c，yc) According to the formulaWherein r is the distance between the center of the eyeball and the edge point of the iris, thereby obtaining the direction coordinate representation of the sight lineand (4) taking the simulated eyeball key point information and the simulated sight direction data as training data to obtain a sight detection model for the convolutional neural network. In the embodiment of the invention, because the complexity in the sight line detection is low, the number of hidden layers of the sight line detection model is set to be smaller than that of the face key point information detection model or the eyeball key point information detection model, for example, the number of hidden layers of the sight line detection model can be set to be smaller than 10, so that the detection efficiency and the detection speed are improved.
according to the technical scheme of the embodiment of the invention, the face area sub-image is obtained by segmenting the face image of the user, so that the sight line detection is more targeted, and the eye sight line of the human eye can be accurately detected. The eye gray level sub-image is subjected to gray level processing to obtain the eye gray level sub-image, so that the complexity of detection is reduced, in addition, the detection is only related to the positions of the pupil and the iris under the scene, the color influence cannot be caused, and the stability of the sight line detection is improved. And carrying out image level mirror image turning processing on the first eye gray level sub-image to obtain a second eye gray level sub-image, and obtaining eyeball key point information according to the two images, thereby improving the sight line detection efficiency.
EXAMPLE III
fig. 5 is a schematic structural diagram of a gaze detection apparatus in a third embodiment of the present invention. The device is suitable for detecting the sight of human eyes, can be realized by software and/or hardware, can be integrated in equipment, and is provided with an image acquisition device for acquiring facial images of a user. Referring to fig. 5, the apparatus specifically includes:
A face key point detection module 310, configured to detect an acquired user face image based on a face detection model to obtain face key point information;
An ocular sub-image determining module 320, configured to determine an ocular sub-image in the user face image according to the facial key point information;
An eyeball key point information detection module 330, configured to determine eyeball key point information according to the ocular subimage based on an eyeball detection model;
and the sight line detection module 340 is configured to determine a sight line detection result according to the eyeball key point information based on a sight line detection model.
optionally, the ocular sub-image determining module 320 includes:
And the face area sub-image determining unit is used for segmenting the user face image according to the face key point information to obtain a face area sub-image.
And the eye sub-image acquisition unit is used for determining the eye sub-image in the face region sub-image according to the eye key point information in the face key point information.
optionally, the method further includes:
The eye gray level sub-image determining module is used for carrying out gray level processing on the eye sub-image to obtain an eye gray level sub-image;
the first acquisition module is used for acquiring an eye region image on any side in the eye gray level sub-image as a first eye gray level sub-image;
the second acquisition module is used for carrying out image level mirror image turning processing on the first eye gray level sub-image to obtain a second eye gray level sub-image;
correspondingly, the eyeball key point information detection module 330 is specifically configured to:
and respectively inputting the first eye gray level sub-image and the second eye gray level sub-image into the eyeball detection model to obtain eyeball key point information output by the eyeball detection model.
optionally, the face detection model is constructed as follows:
Acquiring a sample user face image set, and labeling face key point information on the sample user face image to obtain a labeled user face image set;
and taking the marked user face image set as a training data set, and training the convolutional neural network model to obtain the face detection model.
Optionally, the eyeball detection model is constructed in the following manner:
acquiring a simulated eye subimage and simulated eyeball key point information of a simulated three-dimensional eye model;
And training a convolutional neural network by taking the simulated eye subimage and the simulated eyeball key point information as a training data set to obtain the eyeball detection model.
optionally, the gaze detection model is constructed according to the following method:
Acquiring key point information of a simulated eyeball and simulated sight direction data of a simulated three-dimensional eye model;
and training the convolutional neural network by taking the simulated eyeball key point information and the simulated sight direction data as a training data set to obtain a sight detection model, wherein the number of hidden layers of the sight detection model is less than that of the face key point information detection model or the eyeball key point information detection model.
Optionally, the eye keypoint information includes: coordinates of positions of key points of the eye sockets and coordinates of positions of eyeballs;
the eyeball key point information includes: pupil position point coordinates and iris key point position point coordinates.
According to the technical scheme of the embodiment of the invention, a face key point detection module detects an acquired face image of a user based on a face detection model to obtain face key point information; the eye sub-image determining module determines an eye sub-image in the user face image according to the face key point information; the eyeball key point information detection module determines eyeball key point information according to the eye subimage based on an eyeball detection model; and the sight detection module determines a sight detection result according to the eyeball key point information based on the sight detection model. By the scheme, sight line detection can be realized only through common image acquisition equipment, the requirement on hardware performance is reduced, detection results are accurately obtained based on models with high stability and robustness, the problems of unstable calculated amount and lack of generalization capability caused by artificial feature calculation are solved, and therefore the accuracy and stability of sight line detection are improved.
Example four
Fig. 6 is a schematic structural diagram of an apparatus according to a fourth embodiment of the present invention. FIG. 6 illustrates a block diagram of an exemplary device 412 suitable for use in implementing embodiments of the present invention. The device 412 shown in fig. 6 is only an example and should not impose any limitation on the functionality or scope of use of embodiments of the present invention.
As shown in fig. 6, the apparatus 412 includes: one or more processors 416; the memory 428 is configured to store one or more programs, which when executed by the one or more processors 416, enable the one or more processors 416 to implement the gaze detection method provided by the embodiments of the present invention, including:
Detecting the obtained face image of the user based on a face detection model to obtain face key point information;
Determining an eye sub-image in the user face image according to the face key point information;
Determining eyeball key point information according to the eye subimages based on an eyeball detection model;
And determining a sight detection result according to the eyeball key point information based on a sight detection model.
is expressed in the form of general-purpose equipment. The components of device 412 may include, but are not limited to: one or more processors or processors 416, a system memory 428, and a bus 418 that couples the various system components (including the system memory 428 and the processors 416).
bus 418 represents one or more of any of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, and a processor or local bus using any of a variety of bus architectures. By way of example, such architectures include, but are not limited to, Industry Standard Architecture (ISA) bus, micro-channel architecture (MAC) bus, enhanced ISA bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnect (PCI) bus.
device 412 typically includes a variety of computer system readable storage media. These storage media may be any available storage media that can be accessed by device 412 and includes both volatile and nonvolatile storage media, removable and non-removable storage media.
The system memory 428 may include computer system readable storage media in the form of volatile memory, such as Random Access Memory (RAM)430 and/or cache memory 432. The device 412 may further include other removable/non-removable, volatile/nonvolatile computer system storage media. By way of example only, storage system 434 may be used to read from and write to non-removable, nonvolatile magnetic storage media (not shown in FIG. 6, commonly referred to as a "hard drive"). Although not shown in FIG. 6, a magnetic disk drive for reading from and writing to a removable, nonvolatile magnetic disk (e.g., a "floppy disk") and an optical disk drive for reading from or writing to a removable, nonvolatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical storage medium) may be provided. In these cases, each drive may be connected to bus 418 by one or more data storage media interfaces. Memory 428 can include at least one program product having a set (e.g., at least one) of program modules that are configured to carry out the functions of embodiments of the invention.
A program/utility 440 having a set (at least one) of program modules 442 may be stored, for instance, in memory 428, such program modules 462 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which examples or some combination thereof may comprise an implementation of a network environment. Program modules 462 generally perform the functions and/or methodologies of the described embodiments of the invention.
The device 412 may also communicate with one or more external devices 414 (e.g., keyboard, pointing device, display 426, etc.), with one or more devices that enable a user to interact with the device 412, and/or with any devices (e.g., network card, modem, etc.) that enable the device 412 to communicate with one or more other computing devices. Such communication may occur via input/output (I/O) interfaces 422. Also, the device 412 may communicate with one or more networks (e.g., a Local Area Network (LAN), a Wide Area Network (WAN), and/or a public network, such as the internet) through the network adapter 420. As shown, network adapter 420 communicates with the other modules of device 412 over bus 418. It should be appreciated that although not shown in FIG. 6, other hardware and/or software modules may be used in conjunction with device 412, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, among others.
the processor 416 executes various functional applications and data processing by executing at least one of other programs stored in the system memory 428, for example, to implement a gaze detection method provided by an embodiment of the present invention, including:
Detecting the obtained face image of the user based on a face detection model to obtain face key point information;
Determining an eye sub-image in the user face image according to the face key point information;
determining eyeball key point information according to the eye subimages based on an eyeball detection model;
and determining a sight detection result according to the eyeball key point information based on a sight detection model.
EXAMPLE five
an embodiment of the present invention further provides a storage medium containing computer-executable instructions, which when executed by a computer processor, are configured to perform a gaze detection method:
Detecting the obtained face image of the user based on a face detection model to obtain face key point information;
determining an eye sub-image in the user face image according to the face key point information;
determining eyeball key point information according to the eye subimages based on an eyeball detection model;
And determining a sight detection result according to the eyeball key point information based on a sight detection model.
Computer storage media for embodiments of the present invention can take the form of any combination of one or more computer-readable storage media. The computer readable storage medium may be a computer readable signal storage medium or a computer readable storage medium. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In embodiments of the invention, the computer readable storage medium may be any tangible storage medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
A computer readable signal storage medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal storage medium may also be any computer readable storage medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.
Program code embodied on a computer readable storage medium may be transmitted using any appropriate storage medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
computer program code for carrying out operations for aspects of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or device. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider).
It is to be noted that the foregoing is only illustrative of the preferred embodiments of the present invention and the technical principles employed. It will be understood by those skilled in the art that the present invention is not limited to the particular embodiments described herein, but is capable of various obvious changes, rearrangements and substitutions as will now become apparent to those skilled in the art without departing from the scope of the invention. Therefore, although the present invention has been described in greater detail by the above embodiments, the present invention is not limited to the above embodiments, and may include other equivalent embodiments without departing from the spirit of the present invention, and the scope of the present invention is determined by the scope of the appended claims.
Claims (10)
1. a gaze detection method, the method comprising:
Detecting the obtained face image of the user based on a face detection model to obtain face key point information;
Determining an eye sub-image in the user face image according to the face key point information;
determining eyeball key point information according to the eye subimages based on an eyeball detection model;
and determining a sight detection result according to the eyeball key point information based on a sight detection model.
2. the method of claim 1, wherein determining an ocular sub-image in the user's facial image from the facial keypoint information comprises:
Segmenting the user face image according to the face key point information to obtain a face region sub-image;
And determining the eye sub-image in the face region sub-image according to the eye key point information in the face key point information.
3. the method of claim 1, wherein after determining the ocular sub-image in the user facial image according to the facial keypoint information, further comprising:
carrying out gray processing on the eye sub-image to obtain an eye gray sub-image;
Acquiring an eye region image on any side in the eye gray level sub-image as a first eye gray level sub-image;
Carrying out image level mirror image turning processing on the first eye gray level sub-image to obtain a second eye gray level sub-image;
Correspondingly, the determining eyeball key point information according to the eye subimage based on the eyeball detection model includes:
and respectively inputting the first eye gray level sub-image and the second eye gray level sub-image into the eyeball detection model to obtain eyeball key point information output by the eyeball detection model.
4. The method of claim 1, wherein the face detection model is constructed by:
acquiring a sample user face image set, and labeling face key point information on the sample user face image to obtain a labeled user face image set;
And taking the marked user face image set as a training data set, and training the convolutional neural network model to obtain the face detection model.
5. The method of claim 1, wherein the eye detection model is constructed by:
Acquiring a simulated eye subimage and simulated eyeball key point information of a simulated three-dimensional eye model;
and training a convolutional neural network by taking the simulated eye subimage and the simulated eyeball key point information as a training data set to obtain the eyeball detection model.
6. The method of claim 5, wherein the gaze detection model is constructed according to the following:
Acquiring key point information of a simulated eyeball and simulated sight direction data of a simulated three-dimensional eye model;
and training the convolutional neural network by taking the simulated eyeball key point information and the simulated sight direction data as a training data set to obtain a sight detection model, wherein the number of hidden layers of the sight detection model is less than that of the face key point information detection model or the eyeball key point information detection model.
7. the method according to any one of claims 2 to 6,
The eye key point information includes: coordinates of positions of key points of the eye sockets and coordinates of positions of eyeballs;
The eyeball key point information includes: pupil position point coordinates and iris key point position point coordinates.
8. A gaze detection apparatus, characterized in that the apparatus comprises:
The face key point detection module is used for detecting the acquired face image of the user based on the face detection model to obtain face key point information;
The eye sub-image determining module is used for determining an eye sub-image in the user face image according to the face key point information;
the eyeball key point information detection module is used for determining eyeball key point information according to the eye subimage based on an eyeball detection model;
And the sight detection module is used for determining a sight detection result according to the eyeball key point information based on a sight detection model.
9. an apparatus, characterized in that the apparatus comprises:
one or more processors;
a memory for storing one or more programs;
When executed by the one or more processors, cause the one or more processors to implement a gaze detection method as recited in any of claims 1-7.
10. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out a gaze detection method according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910860197.9A CN110555426A (en) | 2019-09-11 | 2019-09-11 | Sight line detection method, device, equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910860197.9A CN110555426A (en) | 2019-09-11 | 2019-09-11 | Sight line detection method, device, equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110555426A true CN110555426A (en) | 2019-12-10 |
Family
ID=68739919
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910860197.9A Pending CN110555426A (en) | 2019-09-11 | 2019-09-11 | Sight line detection method, device, equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110555426A (en) |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110381368A (en) * | 2019-07-11 | 2019-10-25 | 北京字节跳动网络技术有限公司 | Video cover generation method, device and electronic equipment |
| CN111353429A (en) * | 2020-02-28 | 2020-06-30 | 深圳壹账通智能科技有限公司 | Interest degree method and system based on eyeball turning |
| CN111368717A (en) * | 2020-03-02 | 2020-07-03 | 广州虎牙科技有限公司 | Sight line determining method and device, electronic equipment and computer readable storage medium |
| CN111476151A (en) * | 2020-04-03 | 2020-07-31 | 广州市百果园信息技术有限公司 | Eyeball detection method, device, equipment and storage medium |
| CN111632367A (en) * | 2020-05-18 | 2020-09-08 | 歌尔科技有限公司 | Hand-trip system based on visual guidance and hand-trip response method |
| CN113468956A (en) * | 2021-05-24 | 2021-10-01 | 北京迈格威科技有限公司 | Attention judging method, model training method and corresponding device |
Citations (15)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102830793A (en) * | 2011-06-16 | 2012-12-19 | 北京三星通信技术研究有限公司 | Sight tracking method and sight tracking device |
| CN105094337A (en) * | 2015-08-19 | 2015-11-25 | 华南理工大学 | Three-dimensional gaze estimation method based on irises and pupils |
| CN108229301A (en) * | 2017-11-03 | 2018-06-29 | 北京市商汤科技开发有限公司 | Eyelid line detecting method, device and electronic equipment |
| CN108509890A (en) * | 2018-03-27 | 2018-09-07 | 百度在线网络技术（北京）有限公司 | Method and apparatus for extracting information |
| CN108615014A (en) * | 2018-04-27 | 2018-10-02 | 京东方科技集团股份有限公司 | A kind of detection method of eye state, device, equipment and medium |
| CN108734078A (en) * | 2017-12-14 | 2018-11-02 | 北京市商汤科技开发有限公司 | Image processing method, device, electronic equipment, storage medium and program |
| CN108875524A (en) * | 2018-01-02 | 2018-11-23 | 北京旷视科技有限公司 | Gaze estimation method, device, system and storage medium |
| CN108985210A (en) * | 2018-07-06 | 2018-12-11 | 常州大学 | A kind of Eye-controlling focus method and system based on human eye geometrical characteristic |
| CN108985172A (en) * | 2018-06-15 | 2018-12-11 | 北京七鑫易维信息技术有限公司 | A kind of Eye-controlling focus method, apparatus, equipment and storage medium based on structure light |
| CN109214343A (en) * | 2018-09-14 | 2019-01-15 | 北京字节跳动网络技术有限公司 | Method and apparatus for generating face critical point detection model |
| CN109359575A (en) * | 2018-09-30 | 2019-02-19 | 腾讯科技（深圳）有限公司 | Method for detecting human face, method for processing business, device, terminal and medium |
| CN109359512A (en) * | 2018-08-28 | 2019-02-19 | 深圳壹账通智能科技有限公司 | Eyeball position method for tracing, device, terminal and computer readable storage medium |
| CN109508679A (en) * | 2018-11-19 | 2019-03-22 | 广东工业大学 | Realize method, apparatus, equipment and the storage medium of eyeball three-dimensional eye tracking |
| CN109740491A (en) * | 2018-12-27 | 2019-05-10 | 北京旷视科技有限公司 | A kind of human eye sight recognition methods, device, system and storage medium |
| CN109886107A (en) * | 2019-01-15 | 2019-06-14 | 北京奇艺世纪科技有限公司 | Eyes image processing method, equipment, image processing equipment, medium |
-
2019
- 2019-09-11 CN CN201910860197.9A patent/CN110555426A/en active Pending
Patent Citations (15)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102830793A (en) * | 2011-06-16 | 2012-12-19 | 北京三星通信技术研究有限公司 | Sight tracking method and sight tracking device |
| CN105094337A (en) * | 2015-08-19 | 2015-11-25 | 华南理工大学 | Three-dimensional gaze estimation method based on irises and pupils |
| CN108229301A (en) * | 2017-11-03 | 2018-06-29 | 北京市商汤科技开发有限公司 | Eyelid line detecting method, device and electronic equipment |
| CN108734078A (en) * | 2017-12-14 | 2018-11-02 | 北京市商汤科技开发有限公司 | Image processing method, device, electronic equipment, storage medium and program |
| CN108875524A (en) * | 2018-01-02 | 2018-11-23 | 北京旷视科技有限公司 | Gaze estimation method, device, system and storage medium |
| CN108509890A (en) * | 2018-03-27 | 2018-09-07 | 百度在线网络技术（北京）有限公司 | Method and apparatus for extracting information |
| CN108615014A (en) * | 2018-04-27 | 2018-10-02 | 京东方科技集团股份有限公司 | A kind of detection method of eye state, device, equipment and medium |
| CN108985172A (en) * | 2018-06-15 | 2018-12-11 | 北京七鑫易维信息技术有限公司 | A kind of Eye-controlling focus method, apparatus, equipment and storage medium based on structure light |
| CN108985210A (en) * | 2018-07-06 | 2018-12-11 | 常州大学 | A kind of Eye-controlling focus method and system based on human eye geometrical characteristic |
| CN109359512A (en) * | 2018-08-28 | 2019-02-19 | 深圳壹账通智能科技有限公司 | Eyeball position method for tracing, device, terminal and computer readable storage medium |
| CN109214343A (en) * | 2018-09-14 | 2019-01-15 | 北京字节跳动网络技术有限公司 | Method and apparatus for generating face critical point detection model |
| CN109359575A (en) * | 2018-09-30 | 2019-02-19 | 腾讯科技（深圳）有限公司 | Method for detecting human face, method for processing business, device, terminal and medium |
| CN109508679A (en) * | 2018-11-19 | 2019-03-22 | 广东工业大学 | Realize method, apparatus, equipment and the storage medium of eyeball three-dimensional eye tracking |
| CN109740491A (en) * | 2018-12-27 | 2019-05-10 | 北京旷视科技有限公司 | A kind of human eye sight recognition methods, device, system and storage medium |
| CN109886107A (en) * | 2019-01-15 | 2019-06-14 | 北京奇艺世纪科技有限公司 | Eyes image processing method, equipment, image processing equipment, medium |
Non-Patent Citations (2)
| Title |
|---|
| 左林翼: "基于深度选择性神经网络的视线检测", 《中国优秀博硕士学位论文全文数据库(硕士) 信息科技辑》 * |
| 苏海明等: "使用人眼几何特征的视线追踪方法", 《中国图象图形学报》 * |
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110381368A (en) * | 2019-07-11 | 2019-10-25 | 北京字节跳动网络技术有限公司 | Video cover generation method, device and electronic equipment |
| CN111353429A (en) * | 2020-02-28 | 2020-06-30 | 深圳壹账通智能科技有限公司 | Interest degree method and system based on eyeball turning |
| WO2021169642A1 (en) * | 2020-02-28 | 2021-09-02 | 深圳壹账通智能科技有限公司 | Video-based eyeball turning determination method and system |
| CN111368717A (en) * | 2020-03-02 | 2020-07-03 | 广州虎牙科技有限公司 | Sight line determining method and device, electronic equipment and computer readable storage medium |
| CN111476151A (en) * | 2020-04-03 | 2020-07-31 | 广州市百果园信息技术有限公司 | Eyeball detection method, device, equipment and storage medium |
| CN111476151B (en) * | 2020-04-03 | 2023-02-03 | 广州市百果园信息技术有限公司 | Eyeball detection method, device, equipment and storage medium |
| CN111632367A (en) * | 2020-05-18 | 2020-09-08 | 歌尔科技有限公司 | Hand-trip system based on visual guidance and hand-trip response method |
| CN113468956A (en) * | 2021-05-24 | 2021-10-01 | 北京迈格威科技有限公司 | Attention judging method, model training method and corresponding device |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11403757B2 (en) | Sight line detection method and sight line detection device | |
| CN110555426A (en) | Sight line detection method, device, equipment and storage medium | |
| Memo et al. | Head-mounted gesture controlled interface for human-computer interaction | |
| EP3712805B1 (en) | Gesture recognition method, device, electronic device, and storage medium | |
| WO2019237942A1 (en) | Line-of-sight tracking method and apparatus based on structured light, device, and storage medium | |
| KR102334139B1 (en) | Eye gaze tracking based upon adaptive homography mapping | |
| CN109712234B (en) | Three-dimensional human body model generation method, device, equipment and storage medium | |
| CN107909022B (en) | Video processing method and device, terminal equipment and storage medium | |
| CN114120432A (en) | Online learning attention tracking method based on sight estimation and application thereof | |
| US11789528B1 (en) | On-the-fly calibration for improved on-device eye tracking | |
| CN112446322B (en) | Eyeball characteristic detection method, device, equipment and computer readable storage medium | |
| CN113190109A (en) | Input control method and device of head-mounted display equipment and head-mounted display equipment | |
| CN111589138B (en) | Action prediction method, device, equipment and storage medium | |
| CN116051631A (en) | Light spot labeling method and system | |
| CN114549557A (en) | Portrait segmentation network training method, device, equipment and medium | |
| Scherfgen et al. | Estimating the pose of a medical manikin for haptic augmentation of a virtual patient in mixed reality training | |
| CN113139491A (en) | Video conference control method, system, mobile terminal and storage medium | |
| CN111368668B (en) | Three-dimensional hand recognition method and device, electronic equipment and storage medium | |
| CN109857244B (en) | Gesture recognition method and device, terminal equipment, storage medium and VR glasses | |
| CN116012913A (en) | Model training method, face key point detection method, medium and device | |
| CN113326729B (en) | Multi-mode classroom concentration detection method and device | |
| CN113093907B (en) | Man-machine interaction method, system, equipment and storage medium | |
| CN112862840B (en) | Image segmentation method, device, equipment and medium | |
| CN115019396A (en) | Learning state monitoring method, device, equipment and medium | |
| JP2022128414A (en) | Tracheal intubation positioning method based on deep learning, device, and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20210817 Address after: Room 301-112, floor 3, building 2, No. 18, YANGFANGDIAN Road, Haidian District, Beijing 100089 Applicant after: Beijing Rubu Technology Co.,Ltd. Address before: Room 508-598, Xitian Gezhuang Town Government Office Building, No. 8 Xitong Road, Miyun District Economic Development Zone, Beijing 101500 Applicant before: BEIJING ROOBO TECHNOLOGY Co.,Ltd. |
|
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20191210 |