CN113786189B - Head-eye movement composite capturing method and system based on same image acquisition equipment - Google Patents
Head-eye movement composite capturing method and system based on same image acquisition equipment Download PDFInfo
- Publication number
- CN113786189B CN113786189B CN202110873198.4A CN202110873198A CN113786189B CN 113786189 B CN113786189 B CN 113786189B CN 202110873198 A CN202110873198 A CN 202110873198A CN 113786189 B CN113786189 B CN 113786189B
- Authority
- CN
- China
- Prior art keywords
- eye
- head
- capturing
- optical signal
- movement
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/103—Detecting, measuring or recording devices for testing the shape, pattern, colour, size or movement of the body or parts thereof, for diagnostic purposes
- A61B5/11—Measuring movement of the entire body or parts thereof, e.g. head or hand tremor, mobility of a limb
- A61B5/1126—Measuring movement of the entire body or parts thereof, e.g. head or hand tremor, mobility of a limb using a particular sensing technique
- A61B5/1127—Measuring movement of the entire body or parts thereof, e.g. head or hand tremor, mobility of a limb using a particular sensing technique using markers
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B3/00—Apparatus for testing the eyes; Instruments for examining the eyes
- A61B3/10—Objective types, i.e. instruments for examining the eyes independent of the patients' perceptions or reactions
- A61B3/113—Objective types, i.e. instruments for examining the eyes independent of the patients' perceptions or reactions for determining or recording eye movement
Abstract
The invention provides a head-eye movement composite capturing method and a system based on the same image acquisition equipment, wherein the method comprises the following steps: acquiring a first optical signal through an optical acquisition device; identifying head mark points in the first optical signal, and obtaining a head motion capturing result according to the position and/or shape change of the head mark points; identifying a first eye mark point in the first optical signal, and obtaining an eye motion capturing result according to the position and/or shape change of the first eye mark point; the first optical signal is a visible light signal reflected by the head and eyes of the person to be collected; the head movement and eye movement integrated capturing method and system provided by the invention are used for obtaining the head movement capturing result and the eye movement capturing result through the identification and analysis of the same optical signal, so that the problems of compatibility, instantaneity and integration generated when different devices respectively perform head movement and eye movement capturing are avoided, and the head movement and eye movement integrated capturing method and system with strict synchronization time are provided.
Description
Technical Field
The invention relates to the technical field of composite signal acquisition, in particular to a head-eye movement composite capturing method and system based on the same image acquisition equipment.
Background
A neurodevelopmental disorder is a behavioral and cognitive disorder that occurs during the developmental stage, involving significant difficulties in achieving and performing a particular mental, motor, linguistic or social function. Many mental and behavioral disorders that occur during development are associated with behavioral and cognitive deficits (e.g., schizophrenia, bipolar disorder), but only those that are characterized by neurodevelopmental in the core are included in this group. The etiology of neurodevelopmental disorders is complex and ambiguous in many cases.
In the international disease classification ICD-11, neurological disorders include the following:
6A00 mental retardation;
6a01 developmental speech or language disorder;
6a02 autism spectrum disorder;
6a03 developmental disorder;
6a04 developmental motor coordination disorder;
6a05 attention deficit hyperactivity disorder;
6A06 notch movement disorder;
8A05.0 primary tic or tic disorder;
secondary neurodevelopmental syndrome of 6E 60;
6A0Y other specific neurological disorders;
the 6A0Z neurological disorder is not clear.
Currently, neuropsychological tests in the diagnosis of neurodevelopmental disorders generally employ continuous operation tests (CPT), attention variable Tests (TOVA), reactive/non-reactive tasks (Go/NoGo), wisconsin card classification (WCST), similar pattern Matching (MFFT), and the like. In fact, besides neuropsychological test results, the physiological and behavioral characteristics of the patient in specific contexts (which may be paradigm stimulus, detection/intervention tasks, or even daily learning tasks) also have a good promoting effect on the accurate detection and timely intervention of neurodevelopmental disorders.
In the above test and daily situations, the head movement and eye movement characteristics of the subject also have evaluation values, in the prior art, the acquisition of the head movement and eye movement characteristics is often realized by relying on two different groups of optical acquisition devices, so that very high requirements are put forward on compatibility, instantaneity and integration coordination between the two groups of optical acquisition devices, and meanwhile, the cost is relatively high, so that the method is not beneficial to the real-time monitoring of physiological and behavioral characteristics of patients with nerve development disorder in the home environment, school environment or training intervention institution environment.
Disclosure of Invention
The invention provides a head-eye movement composite capturing method and system based on the same image acquisition equipment, which are used for solving the defects of high compatibility, instantaneity and integration matching requirements among optical acquisition equipment and high cost in the prior art and realizing low-cost and high-efficiency composite capturing of head-eye movement signals.
The invention provides a head-eye movement composite capturing method based on the same image acquisition equipment, which comprises the following steps:
acquiring a first optical signal through an optical acquisition device;
identifying head mark points in the first optical signal, and obtaining a head motion capturing result according to the position and/or shape change of the head mark points;
identifying a first eye mark point in the first optical signal, and obtaining an eye motion capturing result according to the position and/or shape change of the first eye mark point;
the first optical signal is a visible light signal reflected by the head and eyes of the person to be collected; the head mark points are set head characteristic points of the acquired person; the first eye mark points are set eye feature points of the person to be collected.
According to the head-eye movement composite capturing method based on the same image capturing device provided by the invention, after the eye movement capturing result is obtained, the head-eye movement composite capturing method based on the same image capturing device further comprises the following steps:
and correcting the eye motion capture result according to the head motion capture result.
According to the head-movement-eye-movement composite capturing method based on the same image acquisition device, the step of identifying the first eye mark point in the first optical signal and obtaining the eye movement capturing result according to the position and/or shape change of the first eye mark point comprises the following steps:
acquiring a second optical signal by the optical acquisition device; the second optical signal is an infrared signal obtained after the infrared signal source emits and is reflected by eyes of a person to be collected;
identifying a first eye mark point in the first optical signal and a second eye mark point in the second optical signal, and obtaining an eye motion capturing result according to the positions of the first eye mark point, the second eye mark point, the optical acquisition equipment and the infrared signal source;
the second eye mark points are infrared reflection points of eyes of the person to be collected.
According to the head-eye movement composite capturing method based on the same image acquisition equipment, the first eye mark point comprises the pupil center of the person to be acquired; the optical acquisition device is integrated with an infrared signal source.
According to the head movement and eye movement composite capturing method based on the same image acquisition equipment, the step of obtaining an eye movement capturing result according to the positions of the first eye marking point, the second eye marking point, the optical acquisition equipment and the infrared signal source comprises the following steps:
establishing an eyeball projection contour circle by taking a second eye marking point as a circle center and r as a radius, wherein r is the distance from the second eye marking point to any eyeball contour edge point;
summing the first vector and the second vector to obtain an eye motion vector as an eye motion capturing result;
the first vector is opposite to the emitting direction of the infrared signal source and has the size ofWherein h is the distance from the second eye mark point to the pupil center of the person to be collected on the eyeball projection contour circle; the second vector is a vector from the second eye mark point to the pupil center of the person to be collected on the plane of the eyeball projection outline circle.
According to the head-eye movement composite capturing method based on the same image capturing device provided by the invention, after the eye movement capturing result is obtained, the head-eye movement composite capturing method based on the same image capturing device further comprises the following steps:
and obtaining the gazing times, gazing time and sight track of the person to be collected as eye movement tracking results according to the eye movement capturing results from time to time.
According to the head-movement-eye-movement composite capturing method based on the same image acquisition equipment, the head-movement-eye-movement composite capturing method based on the same image acquisition equipment further comprises the following steps:
and obtaining a sight depth result according to the pupil size of the acquired person and the ambient light intensity in the first optical signal.
The invention also provides a head-eye movement composite capturing system based on the same image acquisition equipment, which comprises the following components:
the acquisition module is used for acquiring a first optical signal through the optical acquisition equipment;
the head motion capturing module is used for identifying head mark points in the first optical signal and obtaining a head motion capturing result according to the position and/or shape change of the head mark points;
the eye movement capturing module is used for identifying a first eye marking point in the first optical signal and obtaining an eye movement capturing result according to the position and/or shape change of the first eye marking point;
the first optical signal is a visible light signal reflected by the head and eyes of the person to be collected; the head mark points are set head characteristic points of the acquired person; the first eye mark points are set eye feature points of the person to be collected.
The invention also provides an electronic device, which comprises a memory, a processor and a computer program stored on the memory and capable of running on the processor, wherein the processor realizes the steps of the head-eye movement composite capturing method based on the same image acquisition device when executing the program.
The present invention also provides a non-transitory computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of a head-eye movement composite capture method based on the same image acquisition device as any one of the above.
The head movement and eye movement composite capturing method and system based on the same image acquisition equipment provided by the invention can obtain the head movement capturing result and the eye movement capturing result through the same optical signal and the identification analysis, so that the compatibility, the instantaneity and the integration problems generated when different equipment respectively perform head movement and eye movement capturing are avoided, and the head movement and eye movement integrated capturing method and system with strictly synchronous time are provided.
Drawings
In order to more clearly illustrate the invention or the technical solutions of the prior art, the following description will briefly explain the drawings used in the embodiments or the description of the prior art, and it is obvious that the drawings in the following description are some embodiments of the invention, and other drawings can be obtained according to the drawings without inventive effort for a person skilled in the art.
FIG. 1 is a schematic flow chart of a head-eye movement composite capturing method based on the same image capturing device;
FIG. 2 is a second flow chart of a method for capturing head movement and eye movement based on the same image capturing device according to the present invention;
FIG. 3 is a schematic diagram of the structure of the head-eye movement composite capturing system based on the same image capturing device;
fig. 4 is a schematic structural diagram of an electronic device provided by the present invention.
Reference numerals:
10: an acquisition module; 20: a head motion capture module; 30: an eye movement capturing module;
410: a processor; 420: a communication interface; 430: a memory;
440: a communication bus.
Detailed Description
For the purpose of making the objects, technical solutions and advantages of the present invention more apparent, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings, and it is apparent that the described embodiments are some embodiments of the present invention, not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
The head-eye movement composite capturing method and system based on the same image capturing device of the present invention are described below with reference to fig. 1 to 3.
As shown in fig. 1, an embodiment of the present invention provides a head-eye movement composite capturing method based on the same image capturing device, including:
step 101, acquiring a first optical signal through an optical acquisition device;
102, identifying head mark points in the first optical signal, and obtaining a head motion capturing result according to the position and/or shape change of the head mark points;
step 103, identifying a first eye mark point in the first optical signal, and obtaining an eye motion capturing result according to the position and/or shape change of the first eye mark point;
the first optical signal is a visible light signal reflected by the head and eyes of the person to be collected; the head mark points are set head characteristic points of the acquired person; the first eye mark points are set eye feature points of the person to be collected.
It should be noted that the same signal source refers to a signal source acquired by the same or the same group of optical acquisition devices, such as a camera, that is, a first optical signal. Step 102 and step 103 call the first optical signal to respectively realize the recognition and capture of the head motion and the eye motion, and the time sequence of the execution of step 102 and step 103 is not fixed.
In this embodiment, the head mark points may be naturally occurring head feature points of the person to be collected, such as forehead, mastoid, eyes, eyebrows, nose tip, etc.; the mark point can also be worn by the person to be collected, for example, the person to be collected wears a hat, a hair band, a head cover and the like with the reflective mark point, and the reflective mark point can be used as the head mark point.
Similarly, the eye mark points may be naturally occurring eye feature points of the subject, such as pupils; the eye mark point can also be a mark point worn by the person to be collected, for example, the person to be collected wears a contact lens and a pupil with a reflecting mark point, and the reflecting mark point can be used as the eye mark point; the reflection point of the optical signal with specific frequency or wavelength at the specific position of the person to be collected can also be a reflection point collected by the optical collection device when the light signal is irradiated to the eyeball of the user through an external light source.
It should be emphasized that in this embodiment, the first optical signal acquired by the optical acquisition device may be output by the video distributor, so that capturing of the head motion and tracking of the eye motion can be performed synchronously or asynchronously.
In the prior art, a camera is often used as optical acquisition equipment for optical signal acquisition, and the difference between the focus range, resolution, frame rate and focusing condition of the camera aiming at head action and the camera aiming at eye action is larger, so that the compatibility, instantaneity and integration coordination difficulty between the two are higher; meanwhile, the method is limited by the requirements of resolution and frame rate of the motion capture cameras, the cost is often high, and the adoption of two or two groups of motion capture cameras for respectively realizing eye movement and capturing head movement is unfavorable for cost reduction and efficiency improvement.
In this embodiment, the light is converted into an electrical signal on the image sensor through the lens, and after the signal processing, the signal is split: a set of signals flow to an eye movement instrument for eye movement analysis; a set of signal flows to the dynamic capture instrument for capturing head movements.
It should be noted that, in order to realize the head-movement and eye-movement composite capturing based on the same image capturing device, the optical capturing device in this embodiment needs to simultaneously meet the hardware requirement of the head-movement signal capturing and the hardware requirement of the eye-movement signal capturing, in other words, the hardware parameters (such as resolution, frame rate, etc.) of the optical capturing device in this embodiment should be not less than the higher hardware parameter requirements in the head-movement and the eye-movement capturing.
On this basis, since the optical acquisition device of the present embodiment requires a higher resolution for eye motion capture, the occupied space of each frame signal in the first optical signal (compared to the frame signal of lower resolution) is larger; meanwhile, since motion capture (whether head motion capture or eye motion capture) requires a high frame rate for the signal source (typically no less than 100 frames per second), the first optical signal includes a high number of frame signals per second.
Therefore, the frame signal occupies a large space and has a high frame rate, so that the transmission of the first optical signal, especially in the wireless transmission process, has high requirements on bandwidth and transmission rate.
To solve this problem, in a preferred aspect of the present embodiment, the optical acquisition device has a signal preprocessing module integrated therein; the signal preprocessing module is capable of:
preprocessing the first optical signal to obtain a head movement preprocessing signal comprising position and/or shape changes of head mark points;
the first optical signal is preprocessed to obtain an eye movement preprocessing signal comprising a change in position and/or shape of eye marking points.
In this embodiment, the head motion preprocessing signal may be a data pair of a three-dimensional coordinate of a head mark point and a frame sequence number; the eye movement preprocessing signal may be an image acquisition result of a set area around the eye in the first optical signal, or may be an image signal in which an image of the set area around the eye retains an original resolution and the other areas are compressed to reduce the resolution.
The device based on the same image acquisition device means that the head motion capture and the eye motion capture are obtained based on the analysis of the same image signal acquired by the same image acquisition device; the image acquisition device comprises one or more cameras;
in a preferred version of this embodiment, the image acquisition device is constituted by a single camera; the video data collected by the single camera forms a first optical signal for analysis and use in the head motion capturing step and the eye motion capturing step respectively. The source of the analysis data in the head motion capture step can be based on the first optical signal, and other video data can be present to provide better depth of field or smaller blind area, so as to obtain more accurate head motion capture result.
In another preferred aspect of the present embodiment, the image capturing device includes a plurality of cameras, and video data collected by the plurality of cameras is collected to form a first optical signal, so as to be respectively used for analysis in the head motion capturing step and the eye motion capturing step. That is, in the preferred scheme that the N cameras form the image capturing device, the eye motion capturing result is obtained based on the analysis of N groups of videos acquired by all the N cameras, and the head motion capturing result is also obtained based on the analysis of N groups of videos acquired by all the N cameras, instead of grouping the cameras in the image capturing device, a part of the cameras are used for head motion capturing, and the rest of the cameras are used for eye motion capturing.
In the preferred scheme, the video data acquired by each camera in the image acquisition equipment can be respectively used for head motion capture and eye motion capture, and the number of required cameras can be effectively reduced under the same precision, so that the cost is effectively reduced on the premise of ensuring the acquisition accuracy.
For example, as the image capturing apparatus is composed of 2 cameras, in the present preferred embodiment, head motion capturing and eye motion capturing are performed based on video data captured by the 2 cameras, respectively. Compared with the scheme that 1 camera is used for head motion capture and the other 1 camera is used for eye motion capture, the additional 1 camera can provide more accurate depth data and smaller blind area data, and more accurate data can be obtained at the same cost; compared with the scheme that 2 cameras are used for capturing head movements and the other 2 cameras are used for capturing eye movements, half of the cost can be saved under the condition that the capturing result accuracy is equivalent.
The beneficial effects of this embodiment lie in:
according to the head movement and eye movement integrated capturing method, the head movement capturing result and the eye movement capturing result are obtained through identification and analysis of the same optical signal, so that the problems of compatibility, instantaneity and integration generated when different devices respectively perform head movement and eye movement capturing are avoided, and the head movement and eye movement integrated capturing method with strict synchronization time is provided.
According to the above embodiment, the present embodiment provides a head-eye movement composite capturing method based on the same image capturing device, as shown in fig. 2, including:
step 201, acquiring a first optical signal through an optical acquisition device;
step 202, identifying head mark points in the first optical signal, and obtaining a head motion capturing result according to the position and/or shape change of the head mark points;
step 203, identifying a first eye mark point in the first optical signal, and obtaining an eye motion capturing result according to the position and/or shape change of the first eye mark point;
the first optical signal is a visible light signal reflected by the head and eyes of the person to be collected; the head mark points are set head characteristic points of the acquired person; the first eye mark points are set eye feature points of the person to be collected.
In step 204, the eye motion capturing result is corrected based on the head motion capturing result, that is, an eye movement index such as a line of sight is calculated based on the three-dimensional coordinate change of the head and the face.
Step 205, according to the eye motion capturing result from time to time, the gazing times, gazing time and sight line track of the person to be collected are obtained as eye motion tracking results.
And 206, obtaining a sight depth result according to the pupil size of the acquired person and the ambient light intensity in the first optical signal.
The present embodiment further adds steps 204, 205 and 206 based on the previous embodiment, and the execution sequence of these three steps is not fixed, specifically:
step 204 is performed depending on the results of step 202 and step 203, so that step 204 is performed at a later time than step 202 and step 203 in this embodiment.
Step 205 is performed depending on the result of step 203, so that step 205 is performed at a later time than step 203 in the embodiment, but the sequence of steps 204 and 206 is not fixed.
Step 206 may be performed independently at any time after step 201 is completed, and the result of this step may be complementary to the result of step 203, that is, the eye motion capturing result.
The beneficial effects of this embodiment lie in:
the embodiment is based on providing a head-eye movement integrated capturing method with strict synchronization of time:
the eye motion capturing result is corrected through the head motion capturing result, so that a more accurate eye motion capturing result with the head motion influence removed can be provided;
by extracting the quantitative characteristics of the eye motion capture result, a good data basis is provided for the accurate detection and timely intervention of the subsequent possible neural development disorder;
by extracting and quantifying the depth of the line of sight, a more comprehensive data base is further provided for the accurate detection of possible subsequent neural developmental disorders and timely intervention requirements.
According to any of the embodiments described above, in the present embodiment:
the step of identifying the first eye mark point in the first optical signal and obtaining an eye motion capturing result according to the position and/or shape change of the first eye mark point comprises the following steps:
acquiring a second optical signal through an optical acquisition device;
identifying a first eye mark point in the first optical signal and a second eye mark point in the second optical signal, and obtaining an eye motion capturing result according to the positions of the first eye mark point, the second eye mark point, the optical acquisition equipment and the infrared signal source;
the second eye mark points are infrared reflection points of eyes of the person to be collected; the second optical signal is an infrared signal obtained after the infrared signal source emits and is reflected by eyes of a person to be collected.
The first eye mark point comprises the pupil center of the person to be collected; the optical acquisition device is integrated with an infrared signal source.
The step of obtaining an eye motion capturing result according to the positions of the first eye marking point, the second eye marking point, the optical acquisition device and the infrared signal source comprises the following steps:
establishing an eyeball projection contour circle by taking a second eye marking point as a circle center and r as a radius, wherein r is the distance from the second eye marking point to any eyeball contour edge point;
summing the first vector and the second vector to obtain an eye motion vector as an eye motion capturing result;
the first vector is opposite to the emitting direction of the infrared signal source and has the size ofWherein h is the distance from the second eye mark point to the pupil center of the person to be collected on the eyeball projection contour circle; the second vector is a vector from the second eye mark point to the pupil center of the person to be collected on the plane of the eyeball projection outline circle.
The beneficial effects of this embodiment lie in:
according to the embodiment, on the basis of adopting the same group of optical acquisition equipment, the infrared signal source is introduced, so that the accuracy of the eye motion capturing result is further improved, and compared with the traditional scheme of adding the camera, the cost is effectively reduced.
The head-eye movement composite capturing system based on the same image acquisition equipment provided by the invention is described below, and the head-eye movement composite capturing system based on the same image acquisition equipment described below and the head-eye movement composite capturing method based on the same image acquisition equipment described above can be correspondingly referred to each other.
The embodiment of the invention also provides a head-movement and eye-movement composite capturing system based on the same image acquisition equipment, which comprises the following steps:
an acquisition module 1 for acquiring a first optical signal by an optical acquisition device;
the head motion capturing module 2 is used for identifying head mark points in the first optical signal and obtaining a head motion capturing result according to the position and/or shape change of the head mark points;
the eye movement capturing module 3 is configured to identify a first eye mark point in the first optical signal, and obtain an eye movement capturing result according to a position and/or a shape change of the first eye mark point;
the first optical signal is a visible light signal reflected by the head and eyes of the person to be collected; the head mark points are set head characteristic points of the acquired person; the first eye mark points are set eye feature points of the person to be collected.
The beneficial effects of this embodiment lie in:
the head motion capture result and the eye motion capture result are obtained through the same optical signal by recognition analysis, so that the problems of compatibility, instantaneity and integration generated when different devices respectively perform head motion and eye motion capture are avoided, and the head motion and eye motion integrated capture system with strict synchronization time is provided.
Fig. 4 illustrates a physical schematic diagram of an electronic device, as shown in fig. 4, which may include: processor 410, communication interface (Communications Interface) 420, memory 430 and communication bus 440, wherein processor 410, communication interface 420 and memory 430 communicate with each other via communication bus 440. The processor 410 may invoke logic instructions in the memory 430 to perform a head-eye movement composite capture method based on the same image capture device, the method comprising: acquiring a first optical signal through an optical acquisition device; identifying head mark points in the first optical signal, and obtaining a head motion capturing result according to the position and/or shape change of the head mark points; identifying a first eye mark point in the first optical signal, and obtaining an eye motion capturing result according to the position and/or shape change of the first eye mark point; the first optical signal is a visible light signal reflected by the head and eyes of the person to be collected; the head mark points are set head characteristic points of the acquired person; the first eye mark points are set eye feature points of the person to be collected.
Further, the logic instructions in the memory 430 described above may be implemented in the form of software functional units and may be stored in a computer-readable storage medium when sold or used as a stand-alone product. Based on this understanding, the technical solution of the present invention may be embodied essentially or in a part contributing to the prior art or in a part of the technical solution, in the form of a software product stored in a storage medium, comprising several instructions for causing a computer device (which may be a personal computer, a server, a network device, etc.) to perform all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), a magnetic disk, or an optical disk, or other various media capable of storing program codes.
In another aspect, the present invention also provides a computer program product comprising a computer program stored on a non-transitory computer readable storage medium, the computer program comprising program instructions which, when executed by a computer, enable the computer to perform the method of head-eye movement composite capture based on the same image capturing device provided by the above methods, the method comprising: acquiring a first optical signal through an optical acquisition device; identifying head mark points in the first optical signal, and obtaining a head motion capturing result according to the position and/or shape change of the head mark points; identifying a first eye mark point in the first optical signal, and obtaining an eye motion capturing result according to the position and/or shape change of the first eye mark point; the first optical signal is a visible light signal reflected by the head and eyes of the person to be collected; the head mark points are set head characteristic points of the acquired person; the first eye mark points are set eye feature points of the person to be collected.
In yet another aspect, the present invention further provides a non-transitory computer readable storage medium having stored thereon a computer program which, when executed by a processor, is implemented to perform the above-provided head-eye movement composite capturing method based on the same image capturing device, the method comprising: acquiring a first optical signal through an optical acquisition device; identifying head mark points in the first optical signal, and obtaining a head motion capturing result according to the position and/or shape change of the head mark points; identifying a first eye mark point in the first optical signal, and obtaining an eye motion capturing result according to the position and/or shape change of the first eye mark point; the first optical signal is a visible light signal reflected by the head and eyes of the person to be collected; the head mark points are set head characteristic points of the acquired person; the first eye mark points are set eye feature points of the person to be collected.
The apparatus embodiments described above are merely illustrative, wherein the elements illustrated as separate elements may or may not be physically separate, and the elements shown as elements may or may not be physical elements, may be located in one place, or may be distributed over a plurality of network elements. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art will understand and implement the present invention without undue burden.
From the above description of the embodiments, it will be apparent to those skilled in the art that the embodiments may be implemented by means of software plus necessary general hardware platforms, or of course may be implemented by means of hardware. Based on this understanding, the foregoing technical solution may be embodied essentially or in a part contributing to the prior art in the form of a software product, which may be stored in a computer readable storage medium, such as ROM/RAM, a magnetic disk, an optical disk, etc., including several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the method described in the respective embodiments or some parts of the embodiments.
Finally, it should be noted that: the above embodiments are only for illustrating the technical solution of the present invention, and are not limiting; although the invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims (7)
1. The head movement and eye movement composite capturing method based on the same image acquisition equipment is characterized by comprising the following steps of:
acquiring a first optical signal through an optical acquisition device;
identifying head mark points in the first optical signal, and obtaining a head motion capturing result according to the position and/or shape change of the head mark points;
identifying a first eye mark point in the first optical signal, and obtaining an eye motion capturing result according to the position and/or shape change of the first eye mark point;
the first optical signal is a visible light signal reflected by the head and eyes of the person to be collected; the head mark points are set head characteristic points of the acquired person; the first eye mark points are set eye feature points of the person to be collected;
wherein the first eye mark point comprises the pupil center of the person to be collected; the optical acquisition device is integrated with an infrared signal source; the step of identifying the first eye mark point in the first optical signal and obtaining an eye motion capturing result according to the position and/or shape change of the first eye mark point comprises the following steps:
acquiring a second optical signal by the optical acquisition device; the second optical signal is an infrared signal obtained after the infrared signal source emits and is reflected by eyes of a person to be collected;
identifying a first eye mark point in the first optical signal and a second eye mark point in the second optical signal, and obtaining an eye motion capturing result according to the positions of the first eye mark point, the second eye mark point, the optical acquisition equipment and the infrared signal source;
the second eye mark points are infrared reflection points of eyes of the person to be collected;
the step of obtaining an eye motion capturing result according to the positions of the first eye marking point, the second eye marking point, the optical acquisition device and the infrared signal source comprises the following steps:
establishing an eyeball projection contour circle by taking a second eye marking point as a circle center and r as a radius, wherein r is the distance from the second eye marking point to any eyeball contour edge point;
summing the first vector and the second vector to obtain an eye motion vector as an eye motion capturing result;
the first vector is opposite to the emitting direction of the infrared signal source and has the size ofWherein h is the distance from the second eye mark point to the pupil center of the person to be collected on the eyeball projection contour circle; the second vector is a vector from the second eye mark point to the pupil center of the person to be collected on the plane of the eyeball projection outline circle.
2. The method for capturing the head-movement and eye-movement composite based on the same image capturing device according to claim 1, wherein after obtaining the eye movement capturing result, the method for capturing the head-movement and eye-movement composite based on the same image capturing device further comprises:
and correcting the eye motion capture result according to the head motion capture result.
3. The head-eye movement composite capturing method based on the same image capturing apparatus according to any one of claims 1 to 2, wherein after obtaining the eye movement capturing result, the head-eye movement composite capturing method based on the same image capturing apparatus further comprises:
and obtaining the gazing times, gazing time and sight track of the person to be collected as eye movement tracking results according to the eye movement capturing results from time to time.
4. The head-movement-eye movement composite capturing method based on the same image capturing apparatus according to any one of claims 1 to 2, further comprising:
and obtaining a sight depth result according to the pupil size of the acquired person and the ambient light intensity in the first optical signal.
5. A head-eye movement composite capture system based on the same image acquisition device, comprising:
the acquisition module is used for acquiring a first optical signal through the optical acquisition equipment;
the head motion capturing module is used for identifying head mark points in the first optical signal and obtaining a head motion capturing result according to the position and/or shape change of the head mark points;
the eye movement capturing module is used for identifying a first eye marking point in the first optical signal and obtaining an eye movement capturing result according to the position and/or shape change of the first eye marking point;
the first optical signal is a visible light signal reflected by the head and eyes of the person to be collected; the head mark points are set head characteristic points of the acquired person; the first eye mark points are set eye feature points of the person to be collected;
wherein the first eye mark point comprises the pupil center of the person to be collected; the optical acquisition device is integrated with an infrared signal source; the eye movement capturing module is specifically used for:
acquiring a second optical signal by the optical acquisition device; the second optical signal is an infrared signal obtained after the infrared signal source emits and is reflected by eyes of a person to be collected;
identifying a first eye mark point in the first optical signal and a second eye mark point in the second optical signal, and obtaining an eye motion capturing result according to the positions of the first eye mark point, the second eye mark point, the optical acquisition equipment and the infrared signal source;
the second eye mark points are infrared reflection points of eyes of the person to be collected;
the eye movement capturing module is also specifically used for:
establishing an eyeball projection contour circle by taking a second eye marking point as a circle center and r as a radius, wherein r is the distance from the second eye marking point to any eyeball contour edge point;
summing the first vector and the second vector to obtain an eye motion vector as an eye motion capturing result;
the first vector is opposite to the emitting direction of the infrared signal source and has the size ofWherein h is the distance from the second eye mark point to the pupil center of the person to be collected on the eyeball projection contour circle; the second vector is a vector from the second eye mark point to the pupil center of the person to be collected on the plane of the eyeball projection outline circle.
6. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor, when executing the program, performs the steps of the compound head-eye movement capturing method based on the same image capturing device as claimed in any one of claims 1 to 4.
7. A non-transitory computer readable storage medium having stored thereon a computer program, wherein the computer program when executed by a processor implements the steps of the head-eye movement composite capture method based on the same image capturing device as claimed in any one of claims 1 to 4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110873198.4A CN113786189B (en) | 2021-07-30 | 2021-07-30 | Head-eye movement composite capturing method and system based on same image acquisition equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110873198.4A CN113786189B (en) | 2021-07-30 | 2021-07-30 | Head-eye movement composite capturing method and system based on same image acquisition equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113786189A CN113786189A (en) | 2021-12-14 |
| CN113786189B true CN113786189B (en) | 2023-08-01 |
Family
ID=79181463
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110873198.4A Active CN113786189B (en) | 2021-07-30 | 2021-07-30 | Head-eye movement composite capturing method and system based on same image acquisition equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113786189B (en) |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| DE102004020356A1 (en) * | 2004-04-25 | 2005-11-10 | Chronos Vision Gmbh | Eye`s multidimensional movement detecting method for use by e.g. scientist, involves determining movement of eye and head using series of values that are acquired by taking into consideration pupil position and markings at head |
| CN107633206A (en) * | 2017-08-17 | 2018-01-26 | 平安科技（深圳）有限公司 | Eyeball motion capture method, device and storage medium |
Family Cites Families (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| FR2700938B1 (en) * | 1993-01-29 | 1995-04-28 | Centre Nat Rech Scient | Method and device for analyzing the movement of the eye. |
| JPH1066678A (en) * | 1996-08-28 | 1998-03-10 | Nippon Telegr & Teleph Corp <Ntt> | Non-contact line-of-sight measurement device |
| DE19954047B4 (en) * | 1999-10-29 | 2004-08-05 | Chronos Vision Gmbh | Method and arrangement for recording multidimensional eye movements and marking tincture therefor |
| US20070263923A1 (en) * | 2004-04-27 | 2007-11-15 | Gienko Gennady A | Method for Stereoscopic Measuring Image Points and Device for Carrying Out Said Method |
| JP3686418B2 (en) * | 2004-10-29 | 2005-08-24 | アレイ株式会社 | Measuring device and method |
| WO2015073368A1 (en) * | 2013-11-12 | 2015-05-21 | Highland Instruments, Inc. | Analysis suite |
| JP2016158911A (en) * | 2015-03-03 | 2016-09-05 | 株式会社クレッセント | Surgical operation method using image display device, and device using in surgical operation |
-
2021
- 2021-07-30 CN CN202110873198.4A patent/CN113786189B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| DE102004020356A1 (en) * | 2004-04-25 | 2005-11-10 | Chronos Vision Gmbh | Eye`s multidimensional movement detecting method for use by e.g. scientist, involves determining movement of eye and head using series of values that are acquired by taking into consideration pupil position and markings at head |
| CN107633206A (en) * | 2017-08-17 | 2018-01-26 | 平安科技（深圳）有限公司 | Eyeball motion capture method, device and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113786189A (en) | 2021-12-14 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Garbin et al. | Openeds: Open eye dataset | |
| CN111933275A (en) | Depression evaluation system based on eye movement and facial expression | |
| Garbin et al. | Dataset for eye tracking on a virtual reality platform | |
| WO2022129591A1 (en) | System for determining one or more characteristics of a user based on an image of their eye using an ar/vr headset | |
| CN110472546B (en) | Infant non-contact eye movement feature extraction device and method | |
| WO2014146168A1 (en) | Automatic detection of task transition | |
| CN208693251U (en) | A kind of system for accelerating OCT equipment to be aligned with human eye by iris recognition | |
| CN113786189B (en) | Head-eye movement composite capturing method and system based on same image acquisition equipment | |
| KR101850043B1 (en) | Optometry Analyzer based on multi modal sensing data | |
| Parada et al. | ExpertEyes: Open-source, high-definition eyetracking | |
| Huang et al. | An easily compatible eye-tracking system for freely-moving small animals | |
| CN108158555A (en) | A kind of system and method for OCT equipment being accelerated to be aligned with human eye by iris recognition | |
| CN116030042B (en) | Diagnostic device, method, equipment and storage medium for doctor's diagnosis | |
| Leonardo et al. | Impact of generative modeling for fundus image augmentation with improved and degraded quality in the classification of glaucoma | |
| CN115830675B (en) | Gaze point tracking method and device, intelligent glasses and storage medium | |
| CN110879976A (en) | Adaptive intelligent eye movement data processing system and use method thereof | |
| CN113413154A (en) | Wearable eye movement and facial expression synchronous acquisition system | |
| Pinheiro et al. | Automatic ocular version evaluation in images using random forest | |
| CN114067398A (en) | Autistic child communication disorder assisting method based on facial expression recognition | |
| Nedelcu et al. | Smartphone-based Approach for Automatic Focus Assessment in NIR Fundus Images Targeted at Handheld Devices. | |
| CN111419173A (en) | Method and device for measuring macular pigment density based on fundus image | |
| US11503998B1 (en) | Method and a system for detection of eye gaze-pattern abnormalities and related neurological diseases | |
| Jane et al. | A Vision-Based Approach for the Diagnosis of Digital Asthenopia | |
| Lindén | Calibration in deep-learning eye tracking | |
| Sarkar et al. | Video magnification to detect heart rate for drivers |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20230823 Address after: 401329 3rd Floor, Building 1, No. 53 Fengsheng Road, Jinfeng Town, High tech Zone, Jiulongpo District, Chongqing Patentee after: Chongqing Jinsaixing Medical Technology Co.,Ltd. Address before: 201318 floor 3, No. 1, Lane 100, Banxia Road, Pudong New Area, Shanghai Patentee before: Shanghai saizeng Medical Technology Co.,Ltd. |