CN109446892A - Human eye notice positioning method and system based on deep neural network - Google Patents
Human eye notice positioning method and system based on deep neural network Download PDFInfo
- Publication number
- CN109446892A CN109446892A CN201811073698.4A CN201811073698A CN109446892A CN 109446892 A CN109446892 A CN 109446892A CN 201811073698 A CN201811073698 A CN 201811073698A CN 109446892 A CN109446892 A CN 109446892A
- Authority
- CN
- China
- Prior art keywords
- face
- measured
- attention
- distance
- human eye
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/16—Human faces, e.g. facial parts, sketches or expressions
- G06V40/161—Detection; Localisation; Normalisation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/16—Human faces, e.g. facial parts, sketches or expressions
- G06V40/168—Feature extraction; Face representation
- G06V40/171—Local features and components; Facial parts ; Occluding parts, e.g. glasses; Geometrical relationships
Abstract
Human eye notice positioning method provided by the invention based on deep neural network, comprising: crucial point location, the detection of human face posture angle, center point coordinate calculate, and notice that force vector calculates, attention point location.Human eye notice positioning method based on deep neural network of the invention positions to obtain key point by key point, and key point is normalized to obtain corresponding key point coordinate, human face posture angle value is obtained according to key point coordinate, the coordinate of the interpupillary central point of face two to be measured is obtained by calculation, spatial offset distance is obtained by predetermined depth neural network, it is gained attention force vector according to center point coordinate and spatial offset distance, finally judge the intersection point for paying attention to force vector and attention plane whether on attention effective coverage, the positioning result accurate rate of whole process is higher, and it is suitable for different equipment, it is generally applicable under different scenes.
Description
Technical field
The present invention relates to human eye notice positioning fields, more particularly to the human eye notice positioning based on deep neural network
Method and system.
Background technique
Traditional human eye notice positioning be all based on it is double take the photograph, the methods of 3D structure light, TOF determine human eye attention
Position, but such traditional human eye notice positioning method has the disadvantage that 1. accuracys rate are lower and robustness is not strong, and
It may not apply to all scenes, equipment.2. the human eye attention method of different scenes distinct device can not be general.3. tradition camera shooting
Head must strictly be bound together with device screen.To sum up, the accuracy rate of traditional human eye notice positioning method it is lower and
With certain limitation.
Summary of the invention
For overcome the deficiencies in the prior art, one of the objects of the present invention is to provide the human eyes based on deep neural network
Notice positioning method, can solve traditional human eye notice positioning method accuracy rate it is lower and have certain limitation
The problem of.
The second object of the present invention is to provide the human eye notice positioning system based on deep neural network, can solve
The accuracy rate of traditional human eye notice positioning method is lower and has the problem of certain limitation.
The present invention provides the first purpose and is implemented with the following technical solutions:
Human eye notice positioning method based on deep neural network, the human eye notice positioning method are applied to camera shooting
When head acquisition facial image, characterized by comprising:
Crucial point location carries out key point to facial image to be measured by default key point neural network and positions to obtain 68
Key point；
The detection of human face posture angle, is normalized the key point to obtain corresponding key point coordinate, by institute
It states key point coordinate to be input in default human face posture angle detection neural network, the default human face posture angle detection nerve
Network exports human face posture angle value；
Center point coordinate calculates, by establishing the mapping relations of human eye pupil spacing Yu face distance, and according to human eye to be measured
Pupil spacing and the mapping relations calculate face distance to be measured, according to the face to be measured distance, the key point coordinate
And two interpupillary central points in the facial image to be measured are calculated in the known image parameter of the facial image to be measured
Coordinate；
Spatial offset distance calculates, by left eye region image, the right eye region image, face in the facial image to be measured
Area image, default face accounting figure and the human face posture angle value input predetermined depth neural network, the default depth
It spends neural network and exports spatial offset distance；
Pay attention to force vector calculate, according to the spatial offset distance and the center point coordinate be calculated attention to
Amount；
Attention plane and attention effective coverage are marked, mark attention equipment corresponds to the attention in camera axle center
Plane, and attention effective coverage is marked in the attention plane according to attention instrument size；
Attention point location calculates the intersection point for paying attention to force vector and the attention plane, and judges the intersection point
Whether on the attention effective coverage, if so, human eye attention is in attention equipment, if it is not, then human eye attention
Not in attention equipment.
Further, the crucial point location includes:
Image obtains, and obtains the testing image for containing face to be measured；
Face datection detects the facial image to be measured containing face characteristic region in the testing image；
Crucial point location carries out key point to the facial image to be measured by default key point neural network and positions to obtain
68 key points.
Further, the center point coordinate, which calculates, includes:
Mapping relations are established, it is original when being located at default first distance of shaft centers and default second distance of shaft centers by camera acquisition
The face image of face, and obtain corresponding first average pixel value of human eye pupil spacing and the second average picture in the face image
Element value, according to default first distance of shaft centers, default second distance of shaft centers, first average pixel value, described second flat
Equal calculated for pixel values goes out the original mappings relationship of human eye pupil spacing Yu face distance, wherein the face distance is face to taking the photograph
As the distance of head；
Human eye pupil spacing to be measured is generated, image procossing is carried out to the facial image to be measured and obtains human eye pupil spacing to be measured；
Face distance is calculated, face to be measured is gone out according to the original mappings relationship and the human eye pupil distance computation to be measured
Distance；
Coordinate calculates, according to face distance, the key point coordinate and the facial image to be measured to be measured
Know that two interpupillary center point coordinates in the facial image to be measured are calculated in image parameter, wherein the center point coordinate
For the coordinate of two interpupillary central points in the facial image to be measured.
Further, the face image is size of the original face to camera axle center level angle and pitch angle
Range is 0-5 °.
Further, default first distance of shaft centers and default second distance of shaft centers be not identical.
Further, the default human face posture angle detection neural network includes input layer, the first full articulamentum, second
Full articulamentum and output layer.
Further, human face posture angle detection specifically: the key point is normalized to obtain pair
The key point coordinate is entered by input layer, and successively passes through the first full articulamentum and second by the key point coordinate answered
Full articulamentum processing, final output layer export human face posture angle value.
The present invention provides the second purpose and is implemented with the following technical solutions:
Human eye notice positioning system based on deep neural network, characterized by comprising:
Key point locating module, the key point locating module are used for through default key point neural network to face to be measured
Image carries out key point and positions to obtain 68 key points；
Human face posture angle detection module, the human face posture angle detection module are used to carry out normalizing to the key point
Change handles to obtain corresponding key point coordinate, and the key point coordinate is input to default human face posture angle and detects neural network
In, the default human face posture angle detection neural network exports human face posture angle value；
Center point coordinate computing module, the center point coordinate computing module are used for by establishing human eye pupil spacing and face
The mapping relations of distance, and face distance to be measured is calculated according to human eye pupil spacing to be measured and the mapping relations, according to institute
State face to be measured distance, the key point coordinate and the facial image to be measured known image parameter be calculated it is described to
Survey two interpupillary center point coordinates in facial image；
Spatial offset distance calculation module, the spatial offset distance calculation module is used for will be in the facial image to be measured
Left eye region image, right eye region image, human face region image, default face accounting figure and the human face posture angle value
Predetermined depth neural network is inputted, the predetermined depth neural network exports spatial offset distance；
Attention vector calculation module, the attention vector calculation module are used for according to the spatial offset distance and institute
It states center point coordinate and attention force vector is calculated；
Labeling module, the labeling module are used to mark the attention plane that attention equipment corresponds to camera axle center, and
Attention effective coverage is marked in the attention plane according to attention instrument size；
Attention point location module, the attention point location module is for calculating the attention force vector and the attention
The intersection point of power plane, and judge the intersection point whether on the attention effective coverage.
Further, the key point locating module includes camera, Face datection unit and key point positioning unit,
The camera is used to obtain the testing image containing face to be measured；The Face datection unit is for detecting the testing image
In the facial image to be measured containing face characteristic region；The key point positioning unit is used for by presetting key point nerve net
Network carries out key point to the facial image to be measured and positions to obtain 68 key points.
Further, the center point coordinate computing module includes between establishing mapping relations unit, generating human eye pupil to be measured
Away from unit, calculate face distance unit and coordinate calculating unit；
The mapping relations unit of establishing is for being located at default first distance of shaft centers and default second axis by camera acquisition
The heart away from when original face face image, and obtain corresponding first average pixel value of human eye pupil spacing in the face image
With the second average pixel value, the first distance of shaft centers, default second distance of shaft centers, first mean pixel are preset according to described
Value, second average pixel value calculate the original mappings relationship of human eye pupil spacing Yu face distance, wherein the face away from
With a distance from for face to camera；
The generation human eye pupil spacing unit to be measured is used to obtain the facial image progress image procossing to be measured to be measured
Human eye pupil spacing；
The calculating face distance unit is based on according to the original mappings relationship and the human eye pupil spacing to be measured
Calculate face distance to be measured；
The coordinate calculating unit is used for according to face distance, the key point coordinate and the people to be measured to be measured
Two interpupillary center point coordinates in the facial image to be measured are calculated in the known image parameter of face image, wherein described
Center point coordinate is the coordinate of two interpupillary central points in the facial image to be measured.
Compared with prior art, the beneficial effects of the present invention are: the human eye of the invention based on deep neural network pays attention to
Power localization method is positioned to obtain key point by key point, and key point is normalized to obtain corresponding key point
Coordinate obtains human face posture angle value according to key point coordinate, and the interpupillary central point of face two to be measured is obtained by calculation
Coordinate obtains spatial offset distance by predetermined depth neural network, is obtained according to center point coordinate and spatial offset distance
Pay attention to force vector, finally judge the intersection point for paying attention to force vector and attention plane whether on attention effective coverage, if so,
Human eye attention is in attention equipment, if it is not, then human eye attention is not in attention equipment, the positioning result of whole process
Accurate rate is higher, and is suitable for different equipment, generally applicable under different scenes.
The above description is only an overview of the technical scheme of the present invention, in order to better understand the technical means of the present invention,
And can be implemented in accordance with the contents of the specification, the following is a detailed description of the preferred embodiments of the present invention and the accompanying drawings.
A specific embodiment of the invention is shown in detail by following embodiment and its attached drawing.
Detailed description of the invention
The drawings described herein are used to provide a further understanding of the present invention, constitutes part of this application, this hair
Bright illustrative embodiments and their description are used to explain the present invention, and are not constituted improper limitations of the present invention.In the accompanying drawings:
Fig. 1 is the flow diagram of the human eye notice positioning method of the invention based on deep neural network；
Fig. 2 is the module rack composition of the human eye notice positioning system of the invention based on deep neural network.
Specific embodiment
In the following, being described further in conjunction with attached drawing and specific embodiment to the present invention, it should be noted that not
Under the premise of conflicting, new implementation can be formed between various embodiments described below or between each technical characteristic in any combination
Example.
As shown in Figure 1, the human eye notice positioning method of the invention based on deep neural network, comprising the following steps:
Crucial point location carries out key point to facial image to be measured by default key point neural network and positions to obtain 68
Key point；Specifically include: image obtains, and obtains the testing image for containing face to be measured；It is obtained using camera and contains people to be measured
The testing image of face contains face to be measured and other backgrounds in testing image.
Face datection detects the facial image to be measured containing face characteristic region in testing image；It detects above-mentioned to be measured
Face characteristic region in image and the facial image to be measured for obtaining containing only face to be measured.
Crucial point location carries out key point to facial image to be measured by default key point neural network and positions to obtain 68
Key point；The default key point nerve net that can be used is trained to key point neural network input training set in advance
Network is handled to obtain totally 68 key points to facial image to be measured using default key point neural network.
The detection of human face posture angle, is normalized key point to obtain corresponding key point coordinate, by key point
Coordinate is input in default human face posture angle detection neural network, is preset human face posture angle detection neural network and is exported face
Attitude angle angle value；Human face posture angle detection neural network is trained using original training set in the present embodiment, through excessive
The default human face posture angle that secondary repetition training can be used detects neural network.Specifically: normalizing is carried out to key point
Change handles to obtain corresponding key point coordinate, i.e., carries out conversion process to the key point in facial image to be measured and finally obtain unification
Two-dimensional coordinate.Key point coordinate is entered by input layer, and successively passes through the first full articulamentum and the second full articulamentum
Processing, final output layer export human face posture angle value.Human face posture angle value includes level angle value, tilt values and bows
Elevation value；Level angle value is the degree value of face phase left-right rotation, and tilt values are the inclined degree value of face, pitch angle
Value is the degree value that people raises or overlooks on the face.In the present embodiment, key point is normalized to obtain corresponding key
Key point coordinate is entered by input layer, and successively handled by the first full articulamentum and the second full articulamentum by point coordinate,
Final output layer exports human face posture angle value.The dimension of input layer is 1*136, the dimension of the first full articulamentum in the present embodiment
Degree is 136*68, and the dimension of the second full articulamentum is 68*3, and the dimension of output layer is 1*3.Default human face posture angle detection mind
Size through network only has 38k.In the present embodiment preset human face posture model inspection neural network to human face posture angle into
1ms is only needed when row detection.
Center point coordinate calculates, by establishing the mapping relations of human eye pupil spacing Yu face distance, and according to human eye to be measured
Pupil spacing and mapping relations calculate face distance to be measured, according to face distance, key point coordinate and face to be measured to be measured
Two interpupillary center point coordinate in facial image to be measured is calculated in the known image parameter of image.It specifically includes:
Mapping relations are established, it is original when being located at default first distance of shaft centers and default second distance of shaft centers by camera acquisition
The face image of face, and obtain corresponding first average pixel value of human eye pupil spacing and the second mean pixel in face image
Value calculates human eye according to default first distance of shaft centers, default second distance of shaft centers, the first average pixel value, the second average pixel value
The original mappings relationship of pupil spacing and face distance, wherein face distance is distance of the face to camera.In the present embodiment
In, specifically:
Space coordinates are established by origin of the axle center of camera, space coordinates include X-axis, Y-axis and Z axis, are preset
First distance of shaft centers is distance (Z-direction) of the original face to camera axle center, and default first distance of shaft centers of order is in the present embodiment
d1；Default second distance of shaft centers is also distance (Z-direction) of the original face to camera axle center, enables default second in the present embodiment
Distance of shaft centers is d2；And d1 ≠ d2；Just collected by camera when being located at default first distance of shaft centers and default second distance of shaft centers at this time
The face image of original face obtains the width and height of face image, and it is corresponding to obtain human eye pupil spacing in face image
First average pixel value and the second average pixel value；First mean pixel is corresponding with default first distance of shaft centers, the second mean pixel
Corresponding with default second distance of shaft centers, enabling the first mean pixel is L1, and enabling the second mean pixel is L2；According to default first axle center
Away from, default second distance of shaft centers, the first average pixel value, the second average pixel value calculate the original of human eye pupil spacing Yu face distance
Beginning mapping relations, shown in specific mapping relations such as formula (1),
D=k* (L-L1)+d1 (1)
Wherein, d is face distance,D1 is default first distance of shaft centers, and d2 is default second distance of shaft centers, and L is
Human eye pupil spacing, wherein 0 < L≤min (W, H), W are the width of the face image of the collected original face of camera, and H is
The height of the face image of the collected original face of camera, L1 are the first mean pixel, and L2 is the second mean pixel.According to
Only d and L is variable in formula known to above-mentioned formula (1), therefore the variable relation of d and L can be obtained, this variable relation is people
The original mappings relationship of eye pupil spacing and face distance.The face image of the present embodiment is to require original face relative to camera shooting
The level angle and pitch angle magnitude range in the axle center of head are 0-5 °, and due to the error operation of reality, this is in actually detected mistake
Allow to receive certain error in journey.
Human eye pupil spacing to be measured is generated, facial image to be measured is acquired by camera, image is carried out to facial image to be measured
Processing obtains human eye pupil spacing to be measured；Generate human eye pupil spacing to be measured specifically by camera acquisition containing face to be measured to
Facial image is surveyed, Face datection processing, key point localization process and human face posture angle calculation are carried out to facial image to be measured
Processing, obtains untreated human eye pupil spacing and level angle to be measured, and level angle to be measured is face to be measured and camera axle center
Level angle calculates human eye pupil spacing to be measured according to untreated human eye pupil spacing and level angle to be measured.It is illustrated below:
Facial image to be measured is acquired by camera, to facial image Face datection to be measured processing, key point localization process
And the processing of human face posture angle calculation, untreated human eye pupil spacing and level angle to be measured are obtained, this season untreated human eye
Pupil spacing is L_temp, and enabling level angle to be measured is Y, and the facial image to be measured obtained at this time is relative to camera axle center in level
There is angle of rotation Y on position, therefore untreated human eye pupil spacing at this time is converted into human eye pupil spacing when being positive face-like state, it will
Untreated human eye pupil spacing and level angle to be measured, which substitute into formula (2), is calculated human eye pupil spacing to be measured, and formula (2) is as follows
It is shown:
Wherein, L1For human eye pupil spacing to be measured, L_temp is untreated human eye pupil spacing, Y level angle to be measured；Formula
(2) in, Y has to be larger than -90 ° and less than 90 °.
Face distance is calculated, face distance to be measured is gone out according to original mappings relationship and human eye pupil distance computation to be measured.Root
According to the mapping relations and the obtained pupil of human spacing to be measured of formula (2) in formula (1), face distance to be measured is calculated, i.e.,
Distance on face to be measured to the Z axis in the axle center of camera.
Coordinate calculates, according to the known image parameter meter of face distance to be measured, key point coordinate and facial image to be measured
Calculation obtains two interpupillary center point coordinate in facial image to be measured, wherein center point coordinate is two pupils in facial image to be measured
The coordinate of central point between hole.Specifically: in the present embodiment, enable the center between two pupil of left and right in facial image to be measured
Point coordinate be P1, then P1 relative to camera axle center coordinate be (x1, y1, z1), the above-mentioned face to be measured being calculated away from
From d be P1 on Z axis with a distance from, i.e. d=z1.P1 can be calculated in face figure to be measured according to the above-mentioned key point coordinate that obtains
As upper coordinate (w1, h1)；The intersection point of plane and Z axis that X-axis where enabling facial image to be measured and Y-axis are formed is P0, then P0
Coordinate is (0,0, z1), and coordinate of the P0 on facial image to be measured is (W/2, H/2) at this time, wherein W, and H is facial image to be measured
Width with height；X1 is calculated according to following formula (3) and formula (4), y1, formula is as follows,
X1=k* (w1-W1/2-L1)+d1 (3)
Y1=k* (h1-H1/2-L1)+d1 (4)
Wherein, the coordinate in the X-axis fastened using camera axle center as the space coordinate of origin is put centered on x1, centered on y1
The coordinate in the Y-axis fastened using camera axle center as the space coordinate of origin is put, whereinD1 is default first
Distance of shaft centers, d2 are default second distance of shaft centers, and L is people's eye pupil spacing, wherein 0 < L≤min (W1,H1), W1For facial image to be measured
Width, H1For the height of facial image to be measured, L1 is the first mean pixel, and L2 is the second mean pixel.According to above-mentioned formula
The x1 and y1 found out can obtain occurrence i.e. (x1, y1, the z1) of center point coordinate.
Spatial offset distance calculates, by left eye region image, the right eye region image, human face region in facial image to be measured
Image, default face accounting figure and human face posture angle value input predetermined depth neural network, and predetermined depth neural network is defeated
Spatial offset distance out；Spatial offset distance at this time is enabled to be converted to vector as (Δ x, Δ y, Δ z).
Notice that force vector calculates, attention force vector is calculated according to spatial offset distance and center point coordinate；It enables and paying attention to
Force vector is V1, then V1=(Δ x-x1, Δ y-y1, Δ z-z1), can according to the above-mentioned center point coordinate (x1, y1, z1) acquired
V1 is acquired, Δ x-x1 is vector of the attention in X-axis at this time, and Δ y-y1 is vector of the attention in Y-axis, and Δ z-z1 is attention
Vector on Z axis.
Attention plane and attention effective coverage are marked, mark attention equipment corresponds to the attention in camera axle center
Plane, and attention effective coverage is marked in attention plane according to attention instrument size；Specifically: in the present embodiment
Attention equipment is (such as the painting and calligraphy of object under screen or equipment or scene.Spectacular etc.), mark attention equipment first is opposite
The space plane in camera axle center (center of circle of 3 d space coordinate system), i.e. attention plane, specifically: if attention equipment
It is regular planar, such as screen, planar device etc., three not conllinear point p1, p2, p3 can be taken in the plane, calculates each
Space coordinate of the point relative to camera axle center.If attention equipment is concave plane, approximation three can be taken in the plane
A not conllinear point p1, p2, p3, calculate space coordinate of each point relative to camera axle center.Then formed with above three point
Face be attention plane, it is effective that attention is marked in above-mentioned attention plane according to the length and width (size) of attention equipment
Region.
Whether attention point location calculates the intersection point for paying attention to force vector and attention plane, and judges intersection point in attention
On effective coverage, if so, human eye attention is in attention equipment, if it is not, then human eye attention is not in attention equipment.
The intersection point for paying attention to force vector V1 and attention plane is calculated, if having intersection point, and intersection point is on attention effective coverage, then people at this time
The attention force of eye is in attention equipment, conversely, if without intersection point or intersection point not in attention equipment, the attention of human eye
Not in attention equipment.
As shown in Fig. 2, of the present inventionization provides the human eye notice positioning system based on deep neural network, comprising: close
Key point location module, key point locating module are used to carry out key point to facial image to be measured by default key point neural network
Positioning obtains 68 key points；
Human face posture angle detection module, human face posture angle detection module is for being normalized key point
To corresponding key point coordinate, key point coordinate is input in default human face posture angle detection neural network, face is preset
Attitude angle detects neural network and exports human face posture angle value；
Center point coordinate computing module, center point coordinate computing module are used for by establishing human eye pupil spacing and face distance
Mapping relations, and face distance to be measured is calculated according to human eye pupil spacing to be measured and mapping relations, according to face to be measured away from
Known image parameter from, key point coordinate and facial image to be measured be calculated in facial image to be measured two it is interpupillary in
Heart point coordinate；
Spatial offset distance calculation module, spatial offset distance calculation module are used for the left eye area in facial image to be measured
Area image, right eye region image, human face region image, default face accounting figure and human face posture angle value input predetermined depth
Neural network, predetermined depth neural network export spatial offset distance；
Attention vector calculation module, attention vector calculation module are used for according to spatial offset distance and center point coordinate
Attention force vector is calculated；
Labeling module, labeling module for marking the attention plane that attention equipment corresponds to camera axle center, and according to
Attention instrument size marks attention effective coverage in attention plane；
Attention point location module, attention point location module are used to calculate the friendship for paying attention to force vector and attention plane
Point, and judge intersection point whether on attention effective coverage.
In the present embodiment, key point locating module includes camera, Face datection unit and key point positioning unit,
Camera is used to obtain the testing image containing face to be measured；Face datection unit contains face for detecting in testing image
The facial image to be measured of characteristic area；Key point positioning unit is used for through default key point neural network to facial image to be measured
Key point is carried out to position to obtain 68 key points.Center point coordinate computing module include establish mapping relations unit, generate it is to be measured
Human eye pupil spacing unit calculates face distance unit and coordinate calculating unit；Mapping relations unit is established for passing through camera shooting
The face image of original face when head acquisition is located at default first distance of shaft centers and presets the second distance of shaft centers, and obtain face image
Corresponding first average pixel value of middle human eye pupil spacing and the second average pixel value, according to default first distance of shaft centers, default second
Distance of shaft centers, the first average pixel value, the second average pixel value calculate the original mappings relationship of human eye pupil spacing Yu face distance,
Wherein, face distance is distance of the face to camera；Generate human eye pupil spacing unit to be measured for facial image to be measured into
Row image procossing obtains human eye pupil spacing to be measured；Face distance unit is calculated to be used for according to original mappings relationship and human eye to be measured
Pupil distance computation goes out face distance to be measured；Coordinate calculating unit is used for according to face to be measured distance, key point coordinate and to be measured
Two interpupillary center point coordinate in facial image to be measured is calculated in the known image parameter of facial image, wherein central point
Coordinate is the coordinate of two interpupillary central points in facial image to be measured.
Human eye notice positioning method based on deep neural network of the invention, positions to obtain key by key point
Point, and key point is normalized to obtain corresponding key point coordinate, human face posture angle is obtained according to key point coordinate
Angle value, is obtained by calculation the coordinate of the interpupillary central point of face two to be measured, obtains space by predetermined depth neural network
Offset distance gains attention force vector according to center point coordinate and spatial offset distance, and finally judgement pays attention to force vector and note
Whether the intersection point for power plane of anticipating is on attention effective coverage, if so, human eye attention is in attention equipment, if it is not, then
For human eye attention not in attention equipment, the positioning result accurate rate of whole process is higher, and is suitable for different equipment,
It is generally applicable under different scenes.
More than, only presently preferred embodiments of the present invention is not intended to limit the present invention in any form；All current rows
The those of ordinary skill of industry can be shown in by specification attached drawing and above and swimmingly implement the present invention；But all to be familiar with sheet special
The technical staff of industry without departing from the scope of the present invention, is made a little using disclosed above technology contents
The equivalent variations of variation, modification and evolution is equivalent embodiment of the invention；Meanwhile all substantial technologicals according to the present invention
The variation, modification and evolution etc. of any equivalent variations to the above embodiments, still fall within technical solution of the present invention
Within protection scope.
Claims (10)
1. the human eye notice positioning method based on deep neural network, the human eye notice positioning method is applied to camera
When acquiring facial image, characterized by comprising:
Crucial point location carries out key point to facial image to be measured by default key point neural network and positions to obtain 68 keys
Point；
The detection of human face posture angle, is normalized the key point to obtain corresponding key point coordinate, by the pass
Key point coordinate is input in default human face posture angle detection neural network, and the default human face posture angle detects neural network
Export human face posture angle value；
Center point coordinate calculates, by establishing the mapping relations of human eye pupil spacing Yu face distance, and according between human eye pupil to be measured
Away from and the mapping relations calculate face distance to be measured, according to the face to be measured distance, the key point coordinate and
Two interpupillary center point coordinates in the facial image to be measured are calculated in the known image parameter of the facial image to be measured；
Spatial offset distance calculates, by left eye region image, the right eye region image, human face region in the facial image to be measured
Image, default face accounting figure and the human face posture angle value input predetermined depth neural network, the predetermined depth mind
Spatial offset distance is exported through network；
Notice that force vector calculates, attention force vector is calculated according to the spatial offset distance and the center point coordinate；
Attention plane and attention effective coverage are marked, the attention that mark attention equipment corresponds to camera axle center is flat
Face, and attention effective coverage is marked in the attention plane according to attention instrument size；
Attention point location calculates the intersection point for paying attention to force vector and the attention plane, and whether judges the intersection point
On the attention effective coverage, if so, human eye attention is in attention equipment, if it is not, then human eye attention does not exist
In attention equipment.
2. the human eye notice positioning method based on deep neural network as described in claim 1, it is characterised in that: the pass
Key point location includes:
Image obtains, and obtains the testing image for containing face to be measured；
Face datection detects the facial image to be measured containing face characteristic region in the testing image；
Crucial point location carries out key point to the facial image to be measured by default key point neural network and positions to obtain 68
Key point.
3. the human eye notice positioning method based on deep neural network as described in claim 1, it is characterised in that: in described
Heart point coordinate calculates
Mapping relations are established, original face when being located at default first distance of shaft centers and default second distance of shaft centers by camera acquisition
Face image, and obtain corresponding first average pixel value of human eye pupil spacing and the second mean pixel in the face image
Value is averaged according to default first distance of shaft centers, default second distance of shaft centers, first average pixel value, described second
Calculated for pixel values goes out the original mappings relationship of human eye pupil spacing Yu face distance, wherein the face distance is face to camera shooting
The distance of head；
Human eye pupil spacing to be measured is generated, image procossing is carried out to the facial image to be measured and obtains human eye pupil spacing to be measured；
Calculate face distance, according to the original mappings relationship and the human eye pupil distance computation to be measured go out face to be measured away from
From；
Coordinate calculates, according to the known figure of the face to be measured distance, the key point coordinate and the facial image to be measured
As two interpupillary center point coordinates in the facial image to be measured are calculated in parameter, wherein the center point coordinate is institute
State the coordinate of two interpupillary central points in facial image to be measured.
4. the human eye notice positioning method based on deep neural network as claimed in claim 3, it is characterised in that: it is described just
Face image is that the magnitude range of original face to the camera axle center level angle and pitch angle is 0-5 °.
5. the human eye notice positioning method based on deep neural network as claimed in claim 3, it is characterised in that: described pre-
If the first distance of shaft centers and default second distance of shaft centers be not identical.
6. the human eye notice positioning method based on deep neural network as described in claim 1, it is characterised in that: described pre-
If it includes input layer, the first full articulamentum, the second full articulamentum and output layer that human face posture angle, which detects neural network,.
7. the human eye notice positioning method based on deep neural network as claimed in claim 6, it is characterised in that: the people
Face attitude angle detection specifically: the key point is normalized to obtain corresponding key point coordinate, by the pass
Key point coordinate is entered by input layer, and successively by the first full articulamentum and the second full articulamentum processing, final output layer
Export human face posture angle value.
8. the human eye notice positioning system based on deep neural network, characterized by comprising:
Key point locating module, the key point locating module are used for through default key point neural network to facial image to be measured
Key point is carried out to position to obtain 68 key points；
Human face posture angle detection module, the human face posture angle detection module is for being normalized place to the key point
Reason obtains corresponding key point coordinate, and the key point coordinate is input in default human face posture angle detection neural network,
The default human face posture angle detection neural network exports human face posture angle value；
Center point coordinate computing module, the center point coordinate computing module are used for by establishing human eye pupil spacing and face distance
Mapping relations, and face distance to be measured is calculated according to human eye pupil spacing to be measured and the mapping relations, according to it is described to
The people to be measured is calculated in the known image parameter for surveying face distance, the key point coordinate and the facial image to be measured
Two interpupillary center point coordinate in face image；
Spatial offset distance calculation module, the spatial offset distance calculation module are used for the left side in the facial image to be measured
Vitrea eye area image, right eye region image, human face region image, default face accounting figure and human face posture angle value input
Predetermined depth neural network, the predetermined depth neural network export spatial offset distance；
Attention vector calculation module, the attention vector calculation module be used for according to the spatial offset distance and it is described in
Attention force vector is calculated in heart point coordinate；
Labeling module, the labeling module for marking the attention plane that attention equipment corresponds to camera axle center, and according to
Attention instrument size marks attention effective coverage in the attention plane；
Attention point location module, the attention point location module are flat for calculating the attention force vector and the attention
The intersection point in face, and judge the intersection point whether on the attention effective coverage.
9. the human eye notice positioning system based on deep neural network as claimed in claim 8, it is characterised in that: the pass
Key point location module includes camera, Face datection unit and key point positioning unit, and the camera contains for obtaining
The testing image of face to be measured；The Face datection unit be used to detect in the testing image containing face characteristic region
Facial image to be measured；The key point positioning unit be used for by default key point neural network to the facial image to be measured into
Row key point positions to obtain 68 key points.
10. the human eye notice positioning system based on deep neural network as claimed in claim 8, it is characterised in that: described
Center point coordinate computing module includes establishing mapping relations unit, generation human eye pupil spacing unit to be measured, calculating face distance list
Member and coordinate calculating unit；
The mapping relations unit of establishing is for being located at default first distance of shaft centers and default second distance of shaft centers by camera acquisition
When original face face image, and obtain corresponding first average pixel value of human eye pupil spacing and in the face image
Two average pixel values, according to default first distance of shaft centers, default second distance of shaft centers, first average pixel value, institute
State the original mappings relationship that the second average pixel value calculates human eye pupil spacing Yu face distance, wherein the face distance is
Distance of the face to camera；
It is described to generate human eye pupil spacing unit to be measured for obtaining human eye to be measured to the facial image progress image procossing to be measured
Pupil spacing；
The calculating face distance unit according to the original mappings relationship and the human eye pupil distance computation to be measured for going out
Face distance to be measured；
The coordinate calculating unit is used for according to face distance, the key point coordinate and the face figure to be measured to be measured
Two interpupillary center point coordinates in the facial image to be measured are calculated in the known image parameter of picture, wherein the center
Point coordinate is the coordinate of two interpupillary central points in the facial image to be measured.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811073698.4A CN109446892B (en) | 2018-09-14 | 2018-09-14 | Human eye attention positioning method and system based on deep neural network |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811073698.4A CN109446892B (en) | 2018-09-14 | 2018-09-14 | Human eye attention positioning method and system based on deep neural network |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109446892A true CN109446892A (en) | 2019-03-08 |
| CN109446892B CN109446892B (en) | 2023-03-24 |
Family
ID=65532820
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811073698.4A Active CN109446892B (en) | 2018-09-14 | 2018-09-14 | Human eye attention positioning method and system based on deep neural network |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109446892B (en) |
Cited By (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110191234A (en) * | 2019-06-21 | 2019-08-30 | 中山大学 | It is a kind of based on the intelligent terminal unlocking method for watching point analysis attentively |
| CN110400351A (en) * | 2019-07-30 | 2019-11-01 | 晓智科技（成都）有限公司 | A kind of X-ray front end of emission automatic adjusting method and system |
| CN110543813A (en) * | 2019-07-22 | 2019-12-06 | 深思考人工智能机器人科技（北京）有限公司 | Face image and gaze counting method and system based on scene |
| CN110633664A (en) * | 2019-09-05 | 2019-12-31 | 北京大蛋科技有限公司 | Method and device for tracking attention of user based on face recognition technology |
| CN110781754A (en) * | 2019-09-27 | 2020-02-11 | 精英数智科技股份有限公司 | Method, device and system for intelligent monitoring of manual inspection and storage medium |
| WO2020186883A1 (en) * | 2019-03-18 | 2020-09-24 | 北京市商汤科技开发有限公司 | Methods, devices and apparatuses for gaze area detection and neural network training |
| CN112417949A (en) * | 2020-09-28 | 2021-02-26 | 深圳市艾为智能有限公司 | Network teaching attention monitoring system and method based on vision |
| CN112597823A (en) * | 2020-12-07 | 2021-04-02 | 深延科技(北京)有限公司 | Attention recognition method and device, electronic equipment and storage medium |
| CN112766215A (en) * | 2021-01-29 | 2021-05-07 | 北京字跳网络技术有限公司 | Face fusion method and device, electronic equipment and storage medium |
| CN113011286A (en) * | 2021-03-02 | 2021-06-22 | 重庆邮电大学 | Squint discrimination method and system based on deep neural network regression model of video |
| CN113052064A (en) * | 2021-03-23 | 2021-06-29 | 北京思图场景数据科技服务有限公司 | Attention detection method based on face orientation, facial expression and pupil tracking |
Citations (15)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2007058507A (en) * | 2005-08-24 | 2007-03-08 | Konica Minolta Holdings Inc | Line of sight detecting device |
| CN101584573A (en) * | 2008-05-20 | 2009-11-25 | 丛繁滋 | Interpupillary distance, optical range measuring device |
| US20100195046A1 (en) * | 2009-02-03 | 2010-08-05 | Pixeloptics, Inc. | Multifocal measurement device |
| CN102830793A (en) * | 2011-06-16 | 2012-12-19 | 北京三星通信技术研究有限公司 | Sight tracking method and sight tracking device |
| CN103793719A (en) * | 2014-01-26 | 2014-05-14 | 深圳大学 | Monocular distance-measuring method and system based on human eye positioning |
| CN104766059A (en) * | 2015-04-01 | 2015-07-08 | 上海交通大学 | Rapid and accurate human eye positioning method and sight estimation method based on human eye positioning |
| US20150339589A1 (en) * | 2014-05-21 | 2015-11-26 | Brain Corporation | Apparatus and methods for training robots utilizing gaze-based saliency maps |
| JP2016173313A (en) * | 2015-03-17 | 2016-09-29 | 国立大学法人鳥取大学 | Visual line direction estimation system, visual line direction estimation method and visual line direction estimation program |
| CN106355147A (en) * | 2016-08-26 | 2017-01-25 | 张艳 | Acquiring method and detecting method of live face head pose detection regression apparatus |
| CN106598221A (en) * | 2016-11-17 | 2017-04-26 | 电子科技大学 | Eye key point detection-based 3D sight line direction estimation method |
| CN107345814A (en) * | 2017-07-11 | 2017-11-14 | 海安中科智能制造与信息感知应用研发中心 | A kind of mobile robot visual alignment system and localization method |
| CN107392963A (en) * | 2017-06-28 | 2017-11-24 | 北京航空航天大学 | A kind of imitative hawkeye moving target localization method for soft autonomous air refuelling |
| CN107818310A (en) * | 2017-11-03 | 2018-03-20 | 电子科技大学 | A kind of driver attention's detection method based on sight |
| CN107818305A (en) * | 2017-10-31 | 2018-03-20 | 广东欧珀移动通信有限公司 | Image processing method, device, electronic equipment and computer-readable recording medium |
| CN108288280A (en) * | 2017-12-28 | 2018-07-17 | 杭州宇泛智能科技有限公司 | Dynamic human face recognition methods based on video flowing and device |
-
2018
- 2018-09-14 CN CN201811073698.4A patent/CN109446892B/en active Active
Patent Citations (15)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2007058507A (en) * | 2005-08-24 | 2007-03-08 | Konica Minolta Holdings Inc | Line of sight detecting device |
| CN101584573A (en) * | 2008-05-20 | 2009-11-25 | 丛繁滋 | Interpupillary distance, optical range measuring device |
| US20100195046A1 (en) * | 2009-02-03 | 2010-08-05 | Pixeloptics, Inc. | Multifocal measurement device |
| CN102830793A (en) * | 2011-06-16 | 2012-12-19 | 北京三星通信技术研究有限公司 | Sight tracking method and sight tracking device |
| CN103793719A (en) * | 2014-01-26 | 2014-05-14 | 深圳大学 | Monocular distance-measuring method and system based on human eye positioning |
| US20150339589A1 (en) * | 2014-05-21 | 2015-11-26 | Brain Corporation | Apparatus and methods for training robots utilizing gaze-based saliency maps |
| JP2016173313A (en) * | 2015-03-17 | 2016-09-29 | 国立大学法人鳥取大学 | Visual line direction estimation system, visual line direction estimation method and visual line direction estimation program |
| CN104766059A (en) * | 2015-04-01 | 2015-07-08 | 上海交通大学 | Rapid and accurate human eye positioning method and sight estimation method based on human eye positioning |
| CN106355147A (en) * | 2016-08-26 | 2017-01-25 | 张艳 | Acquiring method and detecting method of live face head pose detection regression apparatus |
| CN106598221A (en) * | 2016-11-17 | 2017-04-26 | 电子科技大学 | Eye key point detection-based 3D sight line direction estimation method |
| CN107392963A (en) * | 2017-06-28 | 2017-11-24 | 北京航空航天大学 | A kind of imitative hawkeye moving target localization method for soft autonomous air refuelling |
| CN107345814A (en) * | 2017-07-11 | 2017-11-14 | 海安中科智能制造与信息感知应用研发中心 | A kind of mobile robot visual alignment system and localization method |
| CN107818305A (en) * | 2017-10-31 | 2018-03-20 | 广东欧珀移动通信有限公司 | Image processing method, device, electronic equipment and computer-readable recording medium |
| CN107818310A (en) * | 2017-11-03 | 2018-03-20 | 电子科技大学 | A kind of driver attention's detection method based on sight |
| CN108288280A (en) * | 2017-12-28 | 2018-07-17 | 杭州宇泛智能科技有限公司 | Dynamic human face recognition methods based on video flowing and device |
Non-Patent Citations (1)
| Title |
|---|
| N.ALSUFYANI ET AL.: "Biometric presentation attack detection using gaze alignment", 《2018 IEEE 4TH INTERNATIONAL CONFERENCE ON IDENTITY》 * |
Cited By (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2020186883A1 (en) * | 2019-03-18 | 2020-09-24 | 北京市商汤科技开发有限公司 | Methods, devices and apparatuses for gaze area detection and neural network training |
| CN110191234A (en) * | 2019-06-21 | 2019-08-30 | 中山大学 | It is a kind of based on the intelligent terminal unlocking method for watching point analysis attentively |
| CN110543813A (en) * | 2019-07-22 | 2019-12-06 | 深思考人工智能机器人科技（北京）有限公司 | Face image and gaze counting method and system based on scene |
| CN110400351A (en) * | 2019-07-30 | 2019-11-01 | 晓智科技（成都）有限公司 | A kind of X-ray front end of emission automatic adjusting method and system |
| CN110633664A (en) * | 2019-09-05 | 2019-12-31 | 北京大蛋科技有限公司 | Method and device for tracking attention of user based on face recognition technology |
| CN110781754A (en) * | 2019-09-27 | 2020-02-11 | 精英数智科技股份有限公司 | Method, device and system for intelligent monitoring of manual inspection and storage medium |
| CN112417949A (en) * | 2020-09-28 | 2021-02-26 | 深圳市艾为智能有限公司 | Network teaching attention monitoring system and method based on vision |
| CN112597823A (en) * | 2020-12-07 | 2021-04-02 | 深延科技(北京)有限公司 | Attention recognition method and device, electronic equipment and storage medium |
| CN112766215A (en) * | 2021-01-29 | 2021-05-07 | 北京字跳网络技术有限公司 | Face fusion method and device, electronic equipment and storage medium |
| CN113011286A (en) * | 2021-03-02 | 2021-06-22 | 重庆邮电大学 | Squint discrimination method and system based on deep neural network regression model of video |
| CN113011286B (en) * | 2021-03-02 | 2022-09-09 | 重庆邮电大学 | Squint discrimination method and system based on deep neural network regression model of video |
| CN113052064A (en) * | 2021-03-23 | 2021-06-29 | 北京思图场景数据科技服务有限公司 | Attention detection method based on face orientation, facial expression and pupil tracking |
| CN113052064B (en) * | 2021-03-23 | 2024-04-02 | 北京思图场景数据科技服务有限公司 | Attention detection method based on face orientation, facial expression and pupil tracking |
Also Published As
| Publication number | Publication date |
|---|---|
| CN109446892B (en) | 2023-03-24 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109446892A (en) | Human eye notice positioning method and system based on deep neural network | |
| CN102156537B (en) | A kind of head pose checkout equipment and method | |
| CN104173054B (en) | Measuring method and measuring device for height of human body based on binocular vision technique | |
| CN104596502B (en) | Object posture measuring method based on CAD model and monocular vision | |
| CN109255813A (en) | A kind of hand-held object pose real-time detection method towards man-machine collaboration | |
| CN105303170B (en) | A kind of gaze estimation method based on human eye feature | |
| CN102369413B (en) | A method for determining the relative position of a first and a second imaging device and devices therefore | |
| CN110142785A (en) | A kind of crusing robot visual servo method based on target detection | |
| CN111414798A (en) | Head posture detection method and system based on RGB-D image | |
| CN108257137A (en) | A kind of angle measurement method and system of the automatic interpretation of view-based access control model hot spot | |
| CN110991266B (en) | Binocular face living body detection method and device | |
| CN107016697B (en) | A kind of height measurement method and device | |
| CN1316416C (en) | Head motion estimation from four feature points | |
| CN104517291B (en) | Pose measuring method based on target coaxial circles feature | |
| CN104766309A (en) | Plane feature point navigation and positioning method and device | |
| CN107688184A (en) | A kind of localization method and system | |
| CN109359537A (en) | Human face posture angle detecting method neural network based and system | |
| CN102075736B (en) | Camera calibration method for high-definition intelligent internet protocol (IP) camera | |
| CN106033614B (en) | A kind of mobile camera motion object detection method under strong parallax | |
| CN108109169A (en) | A kind of position and orientation estimation method, device and robot based on rectangle mark | |
| CN104315998B (en) | Door opening degree judgment method based on depth image and azimuth angle | |
| CN109523595A (en) | A kind of architectural engineering straight line corner angle spacing vision measuring method | |
| CN110378246A (en) | Ground detection method, apparatus, computer readable storage medium and electronic equipment | |
| CN106157322B (en) | A kind of camera installation site scaling method based on plane mirror | |
| CN110065075A (en) | A kind of spatial cell robot external status cognitive method of view-based access control model |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |