CN114415832A - Display and aiming system, method, equipment and terminal for head helmet of fighter - Google Patents
Display and aiming system, method, equipment and terminal for head helmet of fighter Download PDFInfo
- Publication number
- CN114415832A CN114415832A CN202210017624.9A CN202210017624A CN114415832A CN 114415832 A CN114415832 A CN 114415832A CN 202210017624 A CN202210017624 A CN 202210017624A CN 114415832 A CN114415832 A CN 114415832A
- Authority
- CN
- China
- Prior art keywords
- eye
- pilot
- module
- head
- helmet
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 41
- 230000004424 eye movement Effects 0.000 claims abstract description 32
- 230000006870 function Effects 0.000 claims abstract description 10
- 210000001508 eye Anatomy 0.000 claims description 73
- 210000003128 head Anatomy 0.000 claims description 51
- 238000013527 convolutional neural network Methods 0.000 claims description 12
- 238000012545 processing Methods 0.000 claims description 12
- 238000000605 extraction Methods 0.000 claims description 9
- 230000004913 activation Effects 0.000 claims description 8
- 230000009467 reduction Effects 0.000 claims description 7
- 238000013528 artificial neural network Methods 0.000 claims description 6
- 238000004422 calculation algorithm Methods 0.000 claims description 6
- 230000004927 fusion Effects 0.000 claims description 6
- 238000005259 measurement Methods 0.000 claims description 6
- 230000003287 optical effect Effects 0.000 claims description 6
- 238000001514 detection method Methods 0.000 claims description 4
- 238000004590 computer program Methods 0.000 claims description 3
- 238000005520 cutting process Methods 0.000 claims description 3
- 238000009826 distribution Methods 0.000 claims description 3
- 239000011159 matrix material Substances 0.000 claims description 3
- 210000002569 neuron Anatomy 0.000 claims description 3
- 238000010606 normalization Methods 0.000 claims description 3
- 238000005457 optimization Methods 0.000 claims description 3
- 238000011176 pooling Methods 0.000 claims description 3
- 238000007781 pre-processing Methods 0.000 claims description 3
- 230000011218 segmentation Effects 0.000 claims description 3
- 238000005516 engineering process Methods 0.000 abstract description 7
- 230000008569 process Effects 0.000 abstract description 4
- 230000000694 effects Effects 0.000 abstract description 2
- 230000004886 head movement Effects 0.000 abstract description 2
- 210000000554 iris Anatomy 0.000 description 6
- 238000010586 diagram Methods 0.000 description 5
- 230000008859 change Effects 0.000 description 4
- 210000005252 bulbus oculi Anatomy 0.000 description 3
- 230000007547 defect Effects 0.000 description 3
- 230000003993 interaction Effects 0.000 description 3
- 238000003860 storage Methods 0.000 description 3
- 238000012549 training Methods 0.000 description 3
- 230000008901 benefit Effects 0.000 description 2
- 230000001815 facial effect Effects 0.000 description 2
- 238000013507 mapping Methods 0.000 description 2
- ORILYTVJVMAKLC-UHFFFAOYSA-N Adamantane Natural products C1C(C2)CC3CC1CC2C3 ORILYTVJVMAKLC-UHFFFAOYSA-N 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 210000004556 brain Anatomy 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- 210000000720 eyelash Anatomy 0.000 description 1
- 230000004438 eyesight Effects 0.000 description 1
- 239000000835 fiber Substances 0.000 description 1
- 238000005286 illumination Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 230000006996 mental state Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000008092 positive effect Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 238000013519 translation Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
- G06F3/013—Eye tracking input arrangements
-
- F—MECHANICAL ENGINEERING; LIGHTING; HEATING; WEAPONS; BLASTING
- F41—WEAPONS
- F41G—WEAPON SIGHTS; AIMING
- F41G1/00—Sighting devices
- F41G1/46—Sighting devices for particular applications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
- G06F18/2411—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches based on the proximity to a decision surface, e.g. support vector machines
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/25—Fusion techniques
- G06F18/253—Fusion techniques of extracted features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
- G06F3/012—Head tracking input arrangements
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/048—Activation functions
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/80—Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
Abstract
The invention belongs to the technical field of sight aiming of fighters, and discloses a fighter helmet display aiming system, method, equipment and terminal, which comprise the following steps: acquiring an eye image of a pilot; measuring the head attitude of the pilot; extracting eye image feature point information; returning the eye feature point information to the eye movement angle; head pose is spatially integrated with three-dimensional gaze angles. The invention adds the technology of eye tracking to measure and calculate the eye orientation of the pilot, and combines the head posture as the attention or interest target direction of the pilot, thereby realizing the functions of target selection, weapon selection, aiming and the like, solving the problems of incomplete sight tracking range, shielding and interference caused by wearing equipment such as helmets, masks and the like by the pilot of fighters, realizing high-precision aiming effect under large-range free head movement through the convenience and independence of eye tracking, simplifying the calibration process and realizing 'instant use' by taking.
Description
Technical Field
The invention belongs to the technical field of line sight aiming of fighters, and particularly relates to a display aiming system, method, equipment and terminal for a fighter helmet.
Background
Currently, the Helmet Mounted Display System (HMDS) is a device used on a fighter plane to provide information and functions to the pilot, allowing the pilot to control and direct the weapon System and the weapon/sensor depending on the head orientation. The main working principle of the aiming system in the helmet display system is to accurately measure the orientation (pitch angle, yaw angle and roll angle) of a pilot helmet relative to a fuselage. Current helmet display sight systems typically use inertial, optical, electromagnetic, and hybrid sensor methods to accurately measure the position and orientation of the helmet.
However, the mainstream technology directly uses the direction of the helmet rather than the eye sight direction of the pilot to represent the direction of the working target, and neglects the more flexible, more efficient and more free interaction mode of eye movement of the pilot. Compared with the method that the head wearing the helmet needs to be rotated to change the sight target direction by a pilot, the method can change the sight of eyeballs so as to change the target direction more quickly, flexibly and accurately.
Through the above analysis, the problems and defects of the prior art are as follows:
(1) the existing combined helmet display aiming system directly uses the direction of a helmet instead of the sight line direction of eyeballs of a pilot to represent the direction of a working target, and ignores the interaction mode of eye movement of the pilot.
(2) The existing eye movement feature extraction generally adopts a method of geometric assumption and model fitting, and the method is difficult to acquire high-precision eye movement features.
(3) The existing eye tracking method usually needs multiple light sources, multiple cameras and multiple calibration points, the hardware configuration of the system is complex, and the weight of a helmet of a pilot and the physical burden of the pilot are increased.
The difficulty in solving the above problems and defects is:
(1) conventional eye tracking methods helmet goggles will block the view of the external camera towards the driver's face. Under the condition of near-eye, a head-mounted eye movement tracking system is adopted to measure and calculate the high-precision gazing direction of a pilot in real time.
(2) Under the condition of near-to-eye, the problems of uneven illumination, image distortion, eye curtain and eyelash shielding and the like are solved by extracting the eye movement characteristics, and the requirements of high precision and instantaneity are met.
(3) And (4) establishing a high-precision eye movement mapping model, namely mapping the extracted eye movement characteristics to the three-dimensional sight line. The high precision of sight line estimation is guaranteed, and meanwhile, a head-mounted eye movement tracking system without calibration is established by only utilizing a monocular camera.
(4) The head pose and the eye movement angle are in different spatial coordinate systems, and the two should be correctly combined to represent the final target direction.
The significance of solving the problems and the defects is as follows:
(1) the high-precision eye movement characteristic point extraction under the situation of fighter pilots is realized.
(2) The high-precision eye movement angle measurement and calculation under the battle aircraft pilot battle occasion are realized.
(3) Eye movement is faster than head movement, allowing the pilot to change direction of attention in a shorter time.
(4) The helmet system based on eye tracking can also perform real-time online evaluation on the mental state and the like of a pilot through the attention direction of eyeballs of the pilot, the sight moving track, the sight staying condition and the like.
Disclosure of Invention
Aiming at the problems in the prior art, the invention provides a display and aiming system, method, equipment and terminal for a fighter helmet.
The invention is realized in such a way that the display aiming method for the helmet of the fighter comprises the following steps:
step one, acquiring an eye image of a pilot;
measuring the head posture of the pilot;
inputting the eye image obtained in the step one into a stacked hourglass-shaped convolution neural network to extract features;
inputting the eye features in the step three into a support vector regression model to predict the gazing direction;
and step five, performing spatial combination of the head posture and the three-dimensional sight angle.
Further, the acquiring the eye image of the pilot in the first step comprises:
the method comprises the steps of obtaining a face image in real time by using a wide-angle camera under the condition of near-to-eye, obtaining face characteristic points through a face recognition algorithm, and further cutting out an eye image. Under the condition of near-to-eye, the face image shot by the wide-angle camera can be distorted, so that the camera is calibrated by adopting a checkerboard calibration method, and the obtained face image is subjected to anti-distortion operation by using OpenCV.
Further, the measuring the head attitude of the pilot in the second step comprises:
the head pose measurement system may use inertial sensors, optical sensors, or hybrid sensor methods to accurately measure the position and orientation of the helmet. High accuracy head pose measurements are typically achieved using optical sensors. The head pose can also be obtained by a python face recognition algorithm library or by pasting Aruco codes on the head and using an OpenCV detection mode.
Further, the inputting of the eye image in the first step of the third step into the stacked hourglass convolution neural network to extract features comprises:
the first convolution layer in the convolution neural network structure adopts 7 multiplied by 7 convolution kernels, the step length is 1, the extracted feature size of the convolution layer is ((W-F)/S) +1, wherein W is the size of an input image, F is the size of the convolution kernels, and S is the step length; the BN layer performs normalization processing on each neuron, so that the mean value of the characteristic distribution is 0, and the variance is 1; the activation layer uses a ReLU activation function; the hourglass module is a four-order hourglass module and is formed by expanding a first-order hourglass module. The first-order hourglass module consists of four residual error modules, a lifting dimension module and a dimensionality reduction module; the residual error module performs two paths of operation on input, wherein the first path is connected in series through convolution layers with convolution kernel scales of 1 × 1, 3 × 3 and 1 × 1, and a BN layer and a ReLU layer are interleaved; the second path is a skip-level path and only comprises a convolution layer with the convolution kernel scale of 1 multiplied by 1, the step length of all convolution kernels in the two paths is 1, when the boundary is out of range, the convolution kernels in the two paths are filled with 1, and finally the two paths are combined for feature fusion; the first-order hourglass module performs two-path operation on input, wherein the first path is subjected to maximum pooling dimension reduction, then passes through three residual modules, and then is subjected to dimensional increase through binary linear interpolation; and the second path only passes through a residual error module, and finally two paths are combined for feature fusion. The four-order hourglass module is formed by recursively replacing a convolution layer with convolution kernel dimension of 3 multiplied by 3 in the middle of the first-order hourglass module by the first-order hourglass module for four times; and inputting the final output of the four-order hourglass module into a soft-argmax layer, and finding the sub-pixel characteristic point coordinates. The stacked hourglass convolutional neural network is trained by a synthetic eye image that provides coordinate labels including eyelid-sclera boundaries, corneal limbal regions or iris-sclera boundaries, and the angle of the eye; the actual pilot eye image is input to the convolutional neural network, and the model will output n coordinates including the eye curtain, iris, canthus, and eyeball center.
Further, the inputting of the eye feature into the support vector regression model in the fourth step predicts the gaze direction including:
setting the inner and outer canthus as c1,c2Using eye width c1-c2Normalizing all detected coordinates of the mark points, and centering a coordinate system by taking an inner canthus as a coordinate origin;through the center of the iris (u)i0,vi0) Minus eyeball center (u)c,vc) Obtaining a two-dimensional gaze prior; the final feature vector consists of n normalized coordinates extracted by the feature extractor and a 2D gaze direction prior; wherein the normalized coordinates include from an eye curtain edge, an iris edge, and an iris center; inputting 2(n +1) features into the SVR, and directly estimating a pitch angle and a yaw angle of the three-dimensional gazing direction; the SVR model is:
f(x)＝wx+b
the optimization goals of the SVR model are as follows:
s.t.yi-wTxi-b≤∈
w, b are parameters to be learned, yi,xiN samples are trained, and the epsilon is a fitting precision control parameter.
Further, the performing of the spatial combination of the head pose and the three-dimensional gaze angle in the fifth step includes:
head posture
A unit vector uniquely defining a direction measured by a head-motion attitude sensor
Viewing angle estimated from viewing
Converting into a spatial rotation matrix:
since the eye movement angle is relative to the camera on the helmet, which is fixed to the head, the eye movement angle can be considered as an additional rotation of the head pose. Thus, the final gaze unit vector may be expressed as:
expressed as an angle:
Another object of the present invention is to provide a display and aiming system for a helmet of a fighter, which applies the display and aiming method for the helmet of the fighter, the display and aiming system for the helmet of the fighter comprising:
the information acquisition module is used for acquiring information of a pilot, and is configured for acquiring the head posture of the pilot under a cabin coordinate system and acquiring the face image information of the pilot when the pilot wears the helmet.
The feature extraction module is used for detecting eye feature points, is configured for processing face image information, comprises face recognition, face feature point detection, eye segmentation and eye image preprocessing, and sends the obtained eye image into a convolutional neural network to obtain eye feature point information.
And the camera calibration module is used for calibrating the camera, configuring the internal parameters and the external parameters for acquiring the camera and performing anti-distortion processing.
And the eye movement tracking module is used for carrying out sight line estimation and is configured for extracting eye movement angle information from the eye feature points.
And the space combination module is used for performing space combination of the head gesture and the three-dimensional sight angle, and is configured for performing space combination of the head gesture under the cockpit coordinate system and the eye movement angle under the camera coordinate system to obtain a final sight direction under the cockpit coordinate system.
It is a further object of the present invention to provide a computer apparatus comprising a memory and a processor, the memory storing a computer program which, when executed by the processor, causes the processor to perform the fighter helmet display aiming method.
Another object of the present invention is to provide an information data processing terminal for implementing the fighter helmet display aiming system.
By combining all the technical schemes, the invention has the advantages and positive effects that: the display aiming system of the fighter helmet is added with the technology based on eye tracking, measures and calculates the eye orientation of a pilot, and combines the head posture as the attention or interest target direction of the pilot so as to realize the functions of target selection, weapon selection, aiming and the like. The invention solves the problems of incomplete sight tracking range, shielding and interference caused by wearing equipment such as helmets, masks and the like by fighter pilots, realizes high-precision sight tracking effect under large-range head free movement through convenience and independence of eye movement tracking, simplifies the calibration process, and realizes 'instant use' without individual calibration.
The key technology of the new generation advanced fighter combined helmet display aiming system provided by the invention is that on the basis of the prior art, an eye tracking technology is added to measure and calculate the eye orientation of a pilot, and the head posture is combined to serve as the attention or interest target direction of the pilot, so that the functions of target selection, weapon selection, aiming and the like are realized. After the principles, methods and application of the sight line tracking technology are comprehensively researched, aiming at the problems of the existing sight line tracking technology, the invention designs the helmet type sight line tracking system which has simple calibration process, does not limit the freedom degree of a user and can provide the real-time fixation point for the user. Meanwhile, the invention can also effectively explore the processing and processing mechanism of human brain to external information, and has important theoretical value and application value for studying psychology, man-machine interaction, military and the like.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings needed to be used in the embodiments of the present invention will be briefly described below, and it is obvious that the drawings described below are only some embodiments of the present invention, and it is obvious for those skilled in the art that other drawings can be obtained according to the drawings without creative efforts.
Fig. 1 is a diagram of the display aiming method of the helmet of the fighter plane according to the embodiment of the invention.
Fig. 2 is a schematic structural view of a display aiming system of a fighter helmet provided by an embodiment of the invention;
in the figure: 1. an information acquisition module; 2. a feature extraction module; 3. a camera calibration module; 4. an eye tracking module; 5. and (4) a space combination module.
Fig. 3 is a flowchart of the operation of the display aiming system for the helmet of the fighter plane according to the embodiment of the invention.
Fig. 4 is a calibration board of a camera calibration system according to an embodiment of the present invention.
Fig. 5 is a hardware configuration diagram of a helmet provided by an embodiment of the present invention.
Fig. 6 is a schematic diagram of an eye feature extractor network architecture according to an embodiment of the present invention.
Fig. 7 is a schematic diagram of converting the line of sight into a screen landing point according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is further described in detail with reference to the following embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
In view of the problems in the prior art, the present invention provides a display and aiming system, method, device and terminal for a fighter helmet, and the present invention is described in detail below with reference to the accompanying drawings.
As shown in fig. 1, the display aiming method for the helmet of the fighter plane, provided by the embodiment of the invention, comprises the following steps:
s101, acquiring an eye image of a pilot;
s102, measuring the head posture of a pilot;
s103, inputting the eye image obtained in the step one into a stacking hourglass-shaped convolution neural network to extract features;
s104, inputting eye features in the third step into a support vector regression model to predict the gazing direction;
and S105, performing spatial combination of the head posture and the three-dimensional sight angle.
As shown in fig. 2, a display and aiming system for a fighter helmet provided by an embodiment of the invention includes:
the information acquisition module 1 is used for acquiring information of a pilot, and is configured to acquire the head posture of the pilot under a cabin coordinate system and acquire the face image information of the pilot when the pilot wears a helmet.
The feature extraction module 2 is used for detecting eye feature points, is configured to process face image information, and comprises face recognition, face feature point detection, eye segmentation and eye image preprocessing, and sends the obtained eye image to a convolutional neural network to obtain eye feature point information.
And the camera calibration module 3 is used for calibrating the camera, configuring internal parameters and external parameters for acquiring the camera, and performing anti-distortion processing.
And the eye movement tracking module 4 is used for carrying out sight line estimation and is configured for extracting eye movement angle information from the eye feature points.
And the space combination module 5 is used for performing space combination of the head gesture and the three-dimensional sight angle, and is configured for performing space combination of the head gesture under the cabin coordinate system and the eye movement angle under the camera coordinate system to obtain a final sight direction under the cabin coordinate system.
The technical solution of the present invention is further described below with reference to specific examples.
As shown in fig. 1-3, the present invention provides a display aiming method for a helmet of a fighter, which specifically comprises the following steps:
s101, acquiring the eye image of the pilot comprises the following steps:
because the traditional eye tracking method utilizes the fixed camera arranged on the reference surface to capture and calculate the orientation of human eyes, and helmet goggles worn by fighter pilots can shield the view of an external camera facing the face of a driver, the wide-angle camera is used for acquiring the face image in real time under the condition of near-to-eye. And obtaining facial feature points by using a dlib face recognition algorithm through python and OpenCV, wherein the facial feature points at least comprise four eye corner points of a left eye and a right eye, and then cutting out an eye image. Since the face image captured by the wide-angle camera will be distorted in the near-to-eye situation, the camera is calibrated by the checkerboard calibration method, as shown in fig. 4. And performing anti-distortion operation on the acquired face image by using the calibrated internal and external parameters of the camera and using OpenCV.
The measurement of the head attitude of the pilot in step S102 includes:
the head pose measurement system may use inertial sensors, optical sensors, or hybrid sensor methods to accurately measure the position and orientation of the helmet, in this embodiment using gyroscopic sensor acquisition, as shown in fig. 5. The head pose can also be obtained by a python face recognition algorithm library or by detecting Aruco codes. The sensor or the camera is connected with the computer, and the head posture of the pilot is acquired in real time through a driving program and an algorithm.
Inputting the eye image in the first step in the step S103 into the stacked hourglass convolutional neural network to extract features includes:
as shown in fig. 6, the first convolutional layer in the convolutional neural network structure adopts a 7 × 7 convolutional kernel with a step size of 1, and the size of the feature extracted by the convolutional layer is ((W-F)/S) +1, where W is the size of the input image, F is the size of the convolutional kernel, and S is the step size; the BN layer performs normalization processing on each neuron, so that the mean value of the characteristic distribution is 0, and the variance is 1; the activation layer uses a ReLU activation function; the hourglass module is a four-order hourglass module and is formed by expanding a first-order hourglass module. The first-order hourglass module consists of four residual error modules, a lifting dimension module and a dimensionality reduction module; the residual error module performs two paths of operation on input, wherein the first path is connected in series through convolution layers with convolution kernel scales of 1 × 1, 3 × 3 and 1 × 1, and a BN layer and a ReLU layer are interleaved; the second path is a skip-level path and only comprises a convolution layer with the convolution kernel scale of 1 multiplied by 1, the step length of all convolution kernels in the two paths is 1, when the boundary is out of range, the convolution kernels in the two paths are filled with 1, and finally the two paths are combined for feature fusion; the first-order hourglass module performs two-path operation on input, wherein the first path is subjected to maximum pooling dimension reduction, then passes through three residual modules, and then is subjected to dimensional increase through binary linear interpolation; and the second path only passes through a residual error module, and finally two paths are combined for feature fusion. The four-order hourglass module is formed by recursively replacing a convolution layer with convolution kernel dimension of 3 multiplied by 3 in the middle of the first-order hourglass module by the first-order hourglass module for four times; and inputting the final output of the four-order hourglass module into a soft-argmax layer, and finding the sub-pixel characteristic point coordinates. The stacked hourglass convolutional neural network is trained by a synthetic eye image that provides coordinate labels including eyelid-sclera boundaries, corneal limbal regions or iris-sclera boundaries, and the angle of the eye; the real pilot eye image is input to the convolutional neural network, and the model will output 18 coordinates including 8 eye curtains, 8 irises, 1 canthus, and 1 eyeball center. The network loss function is:
wherein h isi(p) is the confidence in pixel p predicted by the net,
a thermodynamic diagram predicted for the network. α is a weight coefficient and is set to 1. The robustness of the eye movement feature extraction model is improved through a data enhancement method. The data enhancement used includes translation, rotation, intensity variation, occlusion, scale variation, and dimension reduction and dimension re-lifting. The model adopts course type learning, and the training noise is increased along with the training. Training Using ADAM optimizer, learning Rate 5, batch size 16, use l2Regularization by a factor of 10-4The ReLU activation function is used. The model was trained on Nvidia 1660Ti GPUs for 600000 epochs until complete or near convergence.
The step S104 of inputting the eye feature into the support vector regression model to predict the gazing direction includes:
setting the inner and outer canthus as c1,c2Using eye width c1-c2Normalizing all detected coordinates of the mark points, and centering a coordinate system by taking an inner canthus as a coordinate origin; through the center of the iris (u)i0,vi0) Minus eyeball center (u)c,vc) Obtaining a two-dimensional gaze prior; the final feature vector consists of n normalized coordinates extracted by the feature extractor and a 2D gaze direction prior; wherein the normalized coordinates comprise 36 features from 8 eye curtain edges, 8 iris edges, 1 iris center and 1 eyeball center; inputting the 36 features into an SVR (singular value representation), and directly estimating a pitch angle and a yaw angle of the three-dimensional staring direction; the SVR model is:
f(x)＝wx+b
the optimization goals of the SVR model are as follows:
s.t.yi-wTxi-b≤∈
w, b are parameters to be learned, yi,xiN samples are trained, and the epsilon is a fitting precision control parameter. The SVR is trained with MPIIGaze and UT Multiview datasets. The original image is collected through the usb camera, the eye movement feature extraction model and the sight line estimation model are coupled together, real-time reasoning is achieved through python and tenserflow, and the human eye gazing angle is directly predicted. The program can reach 60FPS when running on a desktop computer configured as intel i7-10700 and Nvidia 1660Ti GPU, and the real-time requirement is met.
The performing of the spatial combination of the head pose and the three-dimensional gaze angle in step S105 includes:
as shown in FIG. 5, head pose
Measured by a jy901 gyroscope sensor of WitsensorThe screw instrument sensor is installed in the helmet, is connected with a computer through a usb interface, and acquires the head posture in real time through a driving program. Unit vector for uniquely determining one direction of head posture
Viewing angle estimated from viewing
Converting into a spatial rotation matrix:
since the eye movement angle is relative to the camera on the helmet, which is fixed to the head, the eye movement angle can be considered as an additional rotation of the head pose. Thus, the final gaze unit vector may be expressed as:
expressed as an angle:
By the position of the camera above the screen and the ArUco code collecting head on the camera relative to the screen, as shown in fig. 7, the falling point of the sight line on the screen can be represented as:
wherein ZhIndicating the vertical distance of the head from the screen, Xh,YhThe horizontal and vertical coordinates of the head part under the camera coordinate system above the screen are shown.
The pitch angle is expressed in terms of,
indicating the yaw angle. In the case where the head is 60cm from the screen, the accuracy of the falling point of the line of sight on the screen is 1.72 °.
In the above embodiments, the implementation may be wholly or partially realized by software, hardware, firmware, or any combination thereof. When used in whole or in part, can be implemented in a computer program product that includes one or more computer instructions. When loaded or executed on a computer, cause the flow or functions according to embodiments of the invention to occur, in whole or in part. The computer may be a general purpose computer, a special purpose computer, a network of computers, or other programmable device. The computer instructions may be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another, for example, the computer instructions may be transmitted from one website site, computer, server, or data center to another website site, computer, server, or data center via wire (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL), or wireless (e.g., infrared, wireless, microwave, etc.)). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device, such as a server, a data center, etc., that includes one or more of the available media. The usable medium may be a magnetic medium (e.g., floppy Disk, hard Disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., Solid State Disk (SSD)), among others.
The above description is only for the purpose of illustrating the present invention and the appended claims are not to be construed as limiting the scope of the invention, which is intended to cover all modifications, equivalents and improvements that are within the spirit and scope of the invention as defined by the appended claims.
Claims (9)
1. A display aiming method for a helmet of a fighter plane is characterized by comprising the following steps:
step one, acquiring an eye image of a pilot;
measuring the head posture of the pilot;
inputting the eye image obtained in the step one into a stacked hourglass-shaped convolution neural network to extract features;
inputting the eye features in the step three into a support vector regression model to predict the gazing direction;
and step five, performing spatial combination of the head posture and the three-dimensional sight angle.
2. The method as claimed in claim 1, wherein said step one of acquiring eye images of the pilot comprises: the method comprises the steps of obtaining a face image in real time by using a wide-angle camera under the condition of near-to-eye, obtaining face characteristic points through a face recognition algorithm, and further cutting out an eye image.
3. The display aiming method for the helmet of a fighter plane as recited in claim 1, wherein the step two of measuring the head attitude of the pilot comprises: the head pose measurement system may use inertial sensors, optical sensors, or hybrid sensor methods to accurately measure the position and orientation of the helmet.
4. The fighter plane helmet display aiming method of claim 1, wherein inputting the eye image of step one into a stacked hourglass convolutional neural network extraction feature in step three comprises: the stacked hourglass-shaped convolutional neural network is trained by the synthesized eye images, the real eye images of the pilot are input into a first convolutional layer, a BN layer and an activation layer of the convolutional neural network, the result is input into the network formed by stacking the three hourglass modules, and the result is input into a soft-argmax layer to calculate the coordinates of the feature points; the synthetic eye image providing coordinate labels including an eyelid-sclera boundary, a corneal edge region, or an iris-sclera boundary, and an eye angle; the first convolution layer in the convolution neural network structure adopts 7 multiplied by 7 convolution kernels, the step length is 1, the extracted feature size of the convolution layer is ((W-F)/S) +1, wherein W is the size of an input image, F is the size of the convolution kernels, and S is the step length; the BN layer performs normalization processing on each neuron, so that the mean value of the characteristic distribution is 0, and the variance is 1; the activation layer uses a ReLU activation function; the hourglass module is a four-order hourglass module and is formed by expanding a first-order hourglass module. The first-order hourglass module consists of four residual error modules, a lifting dimension module and a dimensionality reduction module; the residual error module performs two paths of operation on input, wherein the first path is connected in series through convolution layers with convolution kernel scales of 1 × 1, 3 × 3 and 1 × 1, and a BN layer and a ReLU layer are interleaved; the second path is a skip-level path and only comprises a convolution layer with the convolution kernel scale of 1 multiplied by 1, the step length of all convolution kernels in the two paths is 1, when the boundary is out of range, the convolution kernels in the two paths are filled with 1, and finally the two paths are combined for feature fusion; the first-order hourglass module performs two-path operation on input, wherein the first path is subjected to maximum pooling dimension reduction, then passes through three residual modules, and then is subjected to dimensional increase through binary linear interpolation; the second path only passes through a residual error module, and finally two paths are combined for feature fusion; the four-order hourglass module is formed by recursively replacing a convolution layer with convolution kernel dimension of 3 multiplied by 3 in the middle of the first-order hourglass module by the first-order hourglass module for four times; and inputting the final output of the four-order hourglass module into a soft-argmax layer, and finding the sub-pixel characteristic point coordinates.
5. The method of claim 1, wherein the eye feature input support vector regression model in step four predicting the gaze direction comprises: normalizing all detected coordinates of the mark points by using the width of eyes, and centering a coordinate system by taking an inner canthus as a coordinate origin; subtracting the eyeball center coordinate from the iris center coordinate to obtain a two-dimensional gaze prior; the final feature vector consists of n normalized coordinates extracted by the feature extractor and a 2D gaze direction prior; wherein the normalized coordinates include from an eye curtain edge, an iris edge, and an iris center; inputting 2(n +1) features into the SVR, and directly estimating a pitch angle and a yaw angle of the three-dimensional gazing direction; the SVR model is:
f(x)＝wx+b
the optimization goals of the SVR model are as follows:
s.t.yi-wTxi-b≤∈
w, b are parameters to be learned, yi,xiN samples are trained, and the epsilon is a fitting precision control parameter.
6. The method as claimed in claim 1, wherein the spatial combination of the head pose and the three-dimensional line of sight angle in step five comprises: head posture
A unit vector uniquely defining a direction measured by a head-motion attitude sensor
Viewing angle estimated from viewing
Converting into a spatial rotation matrix:
since the eye movement angle is relative to the camera on the helmet, which is fixed to the head, the eye movement angle can be considered as an additional rotation of the head pose; thus, the final gaze unit vector is represented as:
expressed as an angle:
7. A fighter helmet display aiming system for implementing the fighter helmet display aiming method according to any one of claims 1-5, wherein the fighter helmet display aiming system comprises:
the information acquisition module is used for acquiring information of a pilot, and is configured for acquiring the head posture of the pilot under a cabin coordinate system and acquiring the face image information of the pilot when the pilot wears a helmet;
the characteristic extraction module is used for detecting eye characteristic points, is configured for processing human face image information and comprises human face identification, human face characteristic point detection, human eye segmentation and human eye image preprocessing, and sends the obtained human eye image to a convolutional neural network to obtain human eye characteristic point information;
the camera calibration module is used for calibrating the camera, configuring internal parameters and external parameters for acquiring the camera and performing anti-distortion processing;
the eye movement tracking module is used for carrying out sight line estimation and is configured for extracting eye movement angle information from the eye feature points;
and the space combination module is used for performing space combination of the head gesture and the three-dimensional sight angle, and is configured for performing space combination of the head gesture under the cockpit coordinate system and the eye movement angle under the camera coordinate system to obtain a final sight direction under the cockpit coordinate system.
8. A computer device comprising a memory and a processor, the memory storing a computer program that, when executed by the processor, causes the processor to perform the method of any of claims 1-5.
9. An information data processing terminal for implementing the fighter helmet display aiming system of claim 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210017624.9A CN114415832B (en) | 2022-01-07 | 2022-01-07 | Fighter aircraft nose helmet display aiming system, method, equipment and terminal |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210017624.9A CN114415832B (en) | 2022-01-07 | 2022-01-07 | Fighter aircraft nose helmet display aiming system, method, equipment and terminal |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114415832A true CN114415832A (en) | 2022-04-29 |
| CN114415832B CN114415832B (en) | 2023-08-04 |
Family
ID=81271741
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210017624.9A Active CN114415832B (en) | 2022-01-07 | 2022-01-07 | Fighter aircraft nose helmet display aiming system, method, equipment and terminal |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114415832B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116708974A (en) * | 2023-08-01 | 2023-09-05 | 清华大学 | Universal camera interference method and system for head-mounted active vision camera |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| GB1578136A (en) * | 1977-06-20 | 1980-11-05 | Hawker Siddeley Aviation Ltd | Helmet-mounted sights |
| CN104280022A (en) * | 2013-07-13 | 2015-01-14 | 哈尔滨点石仿真科技有限公司 | Digital helmet display device tracking system of visual-aided inertial measuring unit |
| CN104714558A (en) * | 2015-03-27 | 2015-06-17 | 北京航空航天大学 | Pilot helmet follow-up demonstration instrument |
| CN107193383A (en) * | 2017-06-13 | 2017-09-22 | 华南师范大学 | A kind of two grades of Eye-controlling focus methods constrained based on facial orientation |
| FR3062938A1 (en) * | 2017-02-14 | 2018-08-17 | Thales | DEVICE AND METHOD FOR REAL TIME LOOK ANALYSIS |
-
2022
- 2022-01-07 CN CN202210017624.9A patent/CN114415832B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| GB1578136A (en) * | 1977-06-20 | 1980-11-05 | Hawker Siddeley Aviation Ltd | Helmet-mounted sights |
| CN104280022A (en) * | 2013-07-13 | 2015-01-14 | 哈尔滨点石仿真科技有限公司 | Digital helmet display device tracking system of visual-aided inertial measuring unit |
| CN104714558A (en) * | 2015-03-27 | 2015-06-17 | 北京航空航天大学 | Pilot helmet follow-up demonstration instrument |
| FR3062938A1 (en) * | 2017-02-14 | 2018-08-17 | Thales | DEVICE AND METHOD FOR REAL TIME LOOK ANALYSIS |
| CN107193383A (en) * | 2017-06-13 | 2017-09-22 | 华南师范大学 | A kind of two grades of Eye-controlling focus methods constrained based on facial orientation |
Non-Patent Citations (1)
| Title |
|---|
| 毛云丰 等: "基于深度神经网络的视线跟踪技术研究", 现代电子技术, no. 16 * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116708974A (en) * | 2023-08-01 | 2023-09-05 | 清华大学 | Universal camera interference method and system for head-mounted active vision camera |
| CN116708974B (en) * | 2023-08-01 | 2023-10-17 | 清华大学 | Universal camera interference method and system for head-mounted active vision camera |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114415832B (en) | 2023-08-04 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107193383B (en) | Secondary sight tracking method based on face orientation constraint | |
| CN106796449B (en) | Sight tracking method and device | |
| CN104978548B (en) | A kind of gaze estimation method and device based on three-dimensional active shape model | |
| US10636193B1 (en) | Generating graphical representation of a user's face and body using a monitoring system included on a head mounted display | |
| JP2019519859A (en) | System and method for performing gaze tracking | |
| US20140218281A1 (en) | Systems and methods for eye gaze determination | |
| CN104809424B (en) | Method for realizing sight tracking based on iris characteristics | |
| CN105094337A (en) | Three-dimensional gaze estimation method based on irises and pupils | |
| Mehrubeoglu et al. | Real-time eye tracking using a smart camera | |
| KR101255219B1 (en) | Method of eye-gaze tracking and system adopting the method | |
| US20220100268A1 (en) | Eye tracking device and a method thereof | |
| Bang et al. | New computer interface combining gaze tracking and brainwave measurements | |
| CN113808160A (en) | Sight direction tracking method and device | |
| CN115482574A (en) | Screen fixation point estimation method, device, medium and equipment based on deep learning | |
| CN111176447A (en) | Augmented reality eye movement interaction method fusing depth network and geometric model | |
| CN114415832B (en) | Fighter aircraft nose helmet display aiming system, method, equipment and terminal | |
| Liu et al. | 3D model-based gaze tracking via iris features with a single camera and a single light source | |
| Chi et al. | 3-D gaze-estimation method using a multi-camera-multi-light-source system | |
| US11435820B1 (en) | Gaze detection pipeline in an artificial reality system | |
| CN113419624B (en) | Eye movement interaction method and device based on head time sequence signal correction | |
| EP4165460A1 (en) | Eye tracking in near-eye displays | |
| Parada et al. | ExpertEyes: Open-source, high-definition eyetracking | |
| Yang et al. | vGaze: Implicit saliency-aware calibration for continuous gaze tracking on mobile devices | |
| Li et al. | An Eye Tracking based aircraft helmet mounted display aiming system | |
| Skowronek et al. | Eye Tracking Using a Smartphone Camera and Deep Learning |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |