CN118038164A - Personality assessment method and system for eye movement video amplification based on camera device - Google Patents
Personality assessment method and system for eye movement video amplification based on camera device Download PDFInfo
- Publication number
- CN118038164A CN118038164A CN202410215888.4A CN202410215888A CN118038164A CN 118038164 A CN118038164 A CN 118038164A CN 202410215888 A CN202410215888 A CN 202410215888A CN 118038164 A CN118038164 A CN 118038164A
- Authority
- CN
- China
- Prior art keywords
- eye
- video
- personality
- eye video
- determining
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 45
- 230000004424 eye movement Effects 0.000 title claims abstract description 28
- 238000003199 nucleic acid amplification method Methods 0.000 title claims abstract description 26
- 230000003321 amplification Effects 0.000 title claims abstract description 24
- 210000001508 eye Anatomy 0.000 claims abstract description 194
- 230000033001 locomotion Effects 0.000 claims abstract description 39
- 238000012549 training Methods 0.000 claims abstract description 31
- 238000011156 evaluation Methods 0.000 claims abstract description 27
- 210000005252 bulbus oculi Anatomy 0.000 claims abstract description 23
- 239000000463 material Substances 0.000 claims abstract description 18
- 230000009471 action Effects 0.000 claims abstract description 8
- 210000001747 pupil Anatomy 0.000 claims description 63
- 235000008694 Humulus lupulus Nutrition 0.000 claims description 18
- 238000004590 computer program Methods 0.000 claims description 17
- 230000001815 facial effect Effects 0.000 claims description 16
- 230000015654 memory Effects 0.000 claims description 14
- 238000010586 diagram Methods 0.000 claims description 11
- 238000006073 displacement reaction Methods 0.000 claims description 9
- 238000000605 extraction Methods 0.000 claims description 4
- 230000008569 process Effects 0.000 abstract description 7
- 238000012360 testing method Methods 0.000 description 10
- 230000006870 function Effects 0.000 description 5
- 230000008859 change Effects 0.000 description 4
- 230000001149 cognitive effect Effects 0.000 description 3
- 238000004891 communication Methods 0.000 description 3
- 230000008878 coupling Effects 0.000 description 3
- 238000010168 coupling process Methods 0.000 description 3
- 238000005859 coupling reaction Methods 0.000 description 3
- 230000000694 effects Effects 0.000 description 3
- 230000002996 emotional effect Effects 0.000 description 3
- 238000012545 processing Methods 0.000 description 3
- 239000013598 vector Substances 0.000 description 3
- 208000019901 Anxiety disease Diseases 0.000 description 2
- 230000036506 anxiety Effects 0.000 description 2
- 230000004397 blinking Effects 0.000 description 2
- 238000004422 calculation algorithm Methods 0.000 description 2
- 230000008451 emotion Effects 0.000 description 2
- 210000000744 eyelid Anatomy 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 210000003205 muscle Anatomy 0.000 description 2
- 230000010344 pupil dilation Effects 0.000 description 2
- 230000004434 saccadic eye movement Effects 0.000 description 2
- 238000006467 substitution reaction Methods 0.000 description 2
- 208000036119 Frailty Diseases 0.000 description 1
- 241001465754 Metazoa Species 0.000 description 1
- 238000013019 agitation Methods 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 206010003549 asthenia Diseases 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000008901 benefit Effects 0.000 description 1
- 230000002860 competitive effect Effects 0.000 description 1
- 239000011365 complex material Substances 0.000 description 1
- 239000012141 concentrate Substances 0.000 description 1
- 230000008602 contraction Effects 0.000 description 1
- 230000003247 decreasing effect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 230000010339 dilation Effects 0.000 description 1
- 235000019800 disodium phosphate Nutrition 0.000 description 1
- 238000003708 edge detection Methods 0.000 description 1
- 230000005284 excitation Effects 0.000 description 1
- 210000004709 eyebrow Anatomy 0.000 description 1
- 210000000720 eyelash Anatomy 0.000 description 1
- 238000003384 imaging method Methods 0.000 description 1
- 230000010365 information processing Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 230000002452 interceptive effect Effects 0.000 description 1
- 208000016339 iris pattern Diseases 0.000 description 1
- 210000004373 mandible Anatomy 0.000 description 1
- 239000003550 marker Substances 0.000 description 1
- 230000036651 mood Effects 0.000 description 1
- 210000000214 mouth Anatomy 0.000 description 1
- 210000005036 nerve Anatomy 0.000 description 1
- 210000001331 nose Anatomy 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000001737 promoting effect Effects 0.000 description 1
- 230000004478 pupil constriction Effects 0.000 description 1
- 238000001303 quality assessment method Methods 0.000 description 1
- 230000010076 replication Effects 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 210000003786 sclera Anatomy 0.000 description 1
- 238000013515 script Methods 0.000 description 1
- 230000035807 sensation Effects 0.000 description 1
- 230000004599 slow eye movement Effects 0.000 description 1
- 230000000638 stimulation Effects 0.000 description 1
- 230000008685 targeting Effects 0.000 description 1
- 238000009966 trimming Methods 0.000 description 1
- 238000012800 visualization Methods 0.000 description 1
Landscapes
- Image Analysis (AREA)
Abstract
The disclosure provides a personality assessment method and a personality assessment system for eye movement video amplification based on a camera device, which belong to the technical field of personality assessment, and the method comprises the following steps: inputting the first eye video into a pre-training amplifying network to amplify eye actions, so as to obtain a second eye video; the first eye video is an eye motion video when a target user reads a material; extracting eye movement characteristics in a second eye video to obtain characteristic pictures of lateral movement of a plurality of eyeballs along with time; and inputting the characteristic pictures into a personality classification network to obtain a personality evaluation result of the target user. According to the personality assessment method and system based on the eye movement video amplification of the camera device, errors in the personality assessment process can be reduced, the personality assessment mode is simpler and more convenient, and the personality assessment result is more accurate.
Description
Technical Field
The disclosure belongs to the technical field of personality assessment, and in particular relates to a personality assessment method and a personality assessment system for eye movement video amplification based on a camera device.
Background
The result of the existing personality assessment method based on questionnaires is unstable: such as the large five-person scale or the Miers-Briggs type index, usually rely on self-reporting questionnaires, but the questionnaire has long response period, and the testee may not have all the patience accurately respond; or is susceptible to a bias, such as a socially desirable bias, to answer in a more socially acceptable manner; or the test subject gives less referential answers due to lack of self-consciousness, resulting in inaccurate test results.
Disclosure of Invention
The disclosure aims to provide a personality assessment method and a personality assessment system for eye movement video amplification based on an imaging device, so as to solve the problems in the prior art.
In a first aspect of an embodiment of the present disclosure, a personality assessment method is provided, including:
Inputting the first eye video into a pre-training amplifying network to amplify eye actions, so as to obtain a second eye video; the first eye video is an eye motion video when a target user reads a material;
Extracting eye movement characteristics in the second eye video to obtain characteristic pictures of lateral movement of a plurality of eyeballs along with time;
and inputting the characteristic pictures into a personality classification network to obtain a personality evaluation result of the target user.
In a second aspect of embodiments of the present disclosure, there is provided a personality assessment system including:
The eye motion amplifying module is used for inputting the first eye video into the pre-training amplifying network to amplify eye motion so as to obtain a second eye video; the first eye video is an eye motion video when a target user reads a material;
The characteristic picture acquisition module is used for extracting eye movement characteristics in the second eye video to obtain characteristic pictures of lateral movement of a plurality of eyeballs along with time;
And the personality evaluation result acquisition module is used for inputting the characteristic pictures into a personality classification network to obtain the personality evaluation result of the target user.
In a third aspect of the disclosed embodiments, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, where the processor implements the steps of the personality assessment method described above when the processor executes the computer program.
In a fourth aspect of the disclosed embodiments, there is provided a computer readable storage medium storing a computer program which, when executed by a processor, implements the steps of the personality assessment method described above.
The personality assessment method and the personality assessment system based on the eye movement video amplification of the camera device provided by the embodiment of the disclosure have the beneficial effects that:
Firstly, compared with a questionnaire survey mode, the personality assessment method provided by the invention is more convenient, the assessment time is short, and a tester is not easily affected by prejudice, so that the test result is more accurate.
Secondly, the method and the device do not select the video of the whole face of the target user to perform personality evaluation, but intercept the eye videos which are more relevant to personality traits in the face videos to perform personality evaluation, so that the test result is more accurate.
Finally, the method inputs the collected first eye video into a pre-training amplifying network to amplify eye actions, so as to obtain a second eye video; and extracting eye movement characteristics in the second eye video to obtain a characteristic picture, and inputting the characteristic picture into a personality classification network to obtain a personality evaluation result of the target user. The method for amplifying the eye motion in the original first eye video can reduce the error of personality assessment, so that the test result is more accurate.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings that are required for the embodiments or the description of the prior art will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present disclosure, and other drawings may be obtained according to these drawings without inventive effort for a person of ordinary skill in the art.
Fig. 1 is a flow chart of a personality evaluation method according to an embodiment of the present disclosure;
FIG. 2 is a training process of a pre-training amplification network provided in an embodiment of the present disclosure;
FIG. 3 is a diagram of a feature image before trimming according to an embodiment of the present disclosure;
FIG. 4 is a diagram of a trimmed feature provided by one embodiment of the disclosure;
FIG. 5 is a block diagram of a personality evaluation system according to an embodiment of the present disclosure;
Fig. 6 is a schematic block diagram of an electronic device provided in an embodiment of the present disclosure.
Detailed Description
In the following description, for purposes of explanation and not limitation, specific details are set forth, such as particular system configurations, techniques, etc. in order to provide a thorough understanding of the disclosed embodiments. However, it will be apparent to one skilled in the art that the present disclosure may be practiced in other embodiments that depart from these specific details. In other instances, detailed descriptions of well-known systems, devices, circuits, and methods are omitted so as not to obscure the description of the present disclosure with unnecessary detail.
For the purposes of promoting an understanding of the principles and advantages of the disclosure, reference will now be made to the embodiments illustrated in the drawings.
Referring to fig. 1, fig. 1 is a flowchart of a personality evaluation method according to an embodiment of the disclosure, where the method includes:
s101: inputting the first eye video into a pre-training amplifying network to amplify eye actions, so as to obtain a second eye video; the first eye video is an eye motion video when the target user reads the material.
In this embodiment, the first eye video is an eye motion video when the target user reads a material. The target users are testers participating in personality assessment, and the testers can participate in the personality assessment test without age limitation, occupation limitation, culture limitation and gender limitation. The eye motion video can be obtained by directly using the camera equipment to collect eye video when a target user reads materials; or acquiring the whole facial video of the target user when the target user reads the material by using the image pickup equipment, then identifying the eye area in the facial video, and cutting out the video of the eye area from the facial video.
The camera device can be a smart phone, a notebook computer, a camera, a desktop computer with an image acquisition function and the like. The reading material of the tester can excite different personality traits, the form of the reading material can be diversified, such as books, articles, reports, poems, novels, scripts and the like, different forms can have different excitation effects on different personality traits, and the proper material form can be selected according to actual conditions. The difficulty of reading the material should be moderate, neither too simple nor too complex. Too simple materials may not motivate deep thinking and insight, while too complex materials may cause difficulties for the reader to understand.
The eye movement video contains changes of eye areas when a tester reads materials, such as anxiety, pupil dilation, blink frequency increase and the like when the tester sees descriptive words expressing dangers; pupil constriction, reduced blink frequency, slow eye movement, etc. may occur when viewing articles in relatively serious scenes.
And inputting the first eye video into a pre-training amplifying network to amplify eye actions, so as to obtain a second eye video.
S102: and extracting the eye movement characteristics in the second eye video to obtain characteristic pictures of the lateral movement of a plurality of eyeballs along with time.
In this embodiment, the second eye video is a video obtained by amplifying the eye movements in the first eye video, and the second eye video contains abundant eye movement features, such as blink times, blink time intervals, eye jump times, eye jump time intervals, eye jump distances, pupil diameter changes, and the like, and these eye movement features can reflect the fatigue degree, concentration degree, emotional state, personality traits, psychological state, and the like of the tester.
Number of blinks, blink interval: the number of blinks may increase during stress or cognitive load and decrease when people calm or concentrate. For example, a highly nervous person may exhibit a higher number of blinks in tension. Number of hops, time interval, distance: eye jumps, i.e. glances, are often related to information processing and attention. For example, people with a high degree of openness may exhibit more frequent glances when exposed to new stimuli, as they tend to explore new experiences. Pupil diameter variation: pupil dilation can be affected by cognitive and emotional factors. For example, an increase in cognitive load, emotional agitation, interest, or appeal may lead to dilation of the pupil.
And extracting the eye movement characteristics in the second eye video to obtain characteristic pictures of the lateral movement of a plurality of eyeballs along with time.
S103: and inputting the characteristic pictures into a personality classification network to obtain a personality evaluation result of the target user.
In this embodiment, personality psychology generally divides personality traits into a wide range of dimensions, with the most widely accepted being the five-factor model, also known as the "large five" personality trait. Comprising the following steps:
(1) Patency: is expressed as imaginative, rich ideas and sensations and mobility. High openness means that a person is creative, maintains an open attitude to new experiences, while low openness means that a person is more traditional and prefers routine.
(2) Responsibility: is presented as targeting autonomy, due duties and achievements. High responsibility may indicate that a person is organized and reliable, and low responsibility may mean that a person is more leisure or less focused on details.
(3) The external orientation: manifested as positive emotion, confidence, social ability. Outward facing persons are generally considered to seek stimulation and liveness, while inward facing persons may be considered to be conservative and orphan.
(4) Affinity: is manifested as being adept at being in mind, tending to be in the mood and collaborating, rather than being suspected and fight against others. High affinity people tend to be friendly and optimistic, while low affinity people may be more competitive.
(5) The nerve mass: manifested as a tendency to easily experience unpleasant emotions, such as anger, anxiety, depression or frailty. High neuro-quality may mean that a person is more emotionally responsive and more susceptible to stress, while low neuro-quality means mood-stable and insensitive. Before the characteristic pictures are input into the personality classification network, the characteristic pictures of the histories of a plurality of users and the psychological scale results are input into the personality classification network, so that the network learns eye movement nuances of target users with different personality traits when reading materials for exciting the different personality traits.
And inputting the characteristic pictures in the test set into a personality classification network, wherein the personality classification network can output the prediction scores or probabilities of the target users on different personality traits. And comparing the predicted score with a real psychological scale result, and calculating the prediction accuracy and recall rate of the personality classification network. And finally, inputting the actual characteristic picture into a personality classification network to obtain a personality evaluation result of the target user.
In this embodiment, the feature images may be further combined with a psychological assessment scale to obtain a psychological assessment result of the target user.
It can be derived from the above that, first, compared with the questionnaire survey mode, the personality assessment method provided by the present disclosure is more convenient, the assessment time is short, and the tester is not easily affected by the bias, so that the test result is more accurate. Secondly, the method and the device do not select the video of the whole face of the target user to perform personality evaluation, but intercept the eye videos which are more relevant to personality traits in the face videos to perform personality evaluation, so that the test result is more accurate. Finally, the method inputs the collected first eye video into a pre-training amplifying network to amplify eye actions, so as to obtain a second eye video; and extracting eye movement characteristics in the second eye video to obtain a characteristic picture, and inputting the characteristic picture into a personality classification network to obtain a personality evaluation result of the target user. The method for amplifying the eye motion in the original first eye video can reduce the error of personality assessment, so that the test result is more accurate.
In one embodiment of the present disclosure, the personality assessment method further includes:
Determining pupil coordinates in the facial video;
and cutting out the video containing the left eye part and the right eye part in the face video by taking the pupil coordinates as the center to obtain a first eye video.
In this embodiment, the reading material may be displayed on a computer display screen, and the eye position of the target user may be located by capturing the facial image of the target user during reading, recording the facial image, and identifying the key points of the target user's face. The center position of the eye is estimated based on the eye position of the target user, thereby determining pupil coordinates. The change of the pupil position is closely related to the action of the eyes, for example, when blinking, the pressure applied by the eyelid to the eyeballs can cause the pupil to slightly move in the positions inside the eyeballs; rapid contraction and relaxation of the muscles inside the eye during eye jump may occur, resulting in a brief movement of the pupil's position inside the eyeball. After the pupil coordinates are determined, the corresponding left-eye and right-eye areas can be found by taking the pupil coordinates as the center, and the video containing the left-eye and right-eye parts in the face video is cut out to obtain a first eye video.
In one embodiment of the present disclosure, determining pupil coordinates in a facial video includes:
and extracting frames from the facial video to obtain a plurality of continuous first frame images, determining coordinates corresponding to left and right eyes in each first frame image, and determining pupil coordinates according to the characteristics of pupils and the coordinates corresponding to the left and right eyes.
In this embodiment, frames are extracted from the face video to obtain a plurality of continuous first frame images, and coordinates corresponding to left and right eyes in each first frame image are determined. The pre-trained model may be used to detect facial markers and keypoints, such as 68 key facial markers using a Dlib pre-trained facial marker detector. These landmarks include points around the eyes, eyebrows, nose, mouth and mandible. Coordinates corresponding to the left and right eyes are found from the key points, the area between the corner points of each eye is checked, and the center of the left eye area and the center of the right eye area are estimated according to the circular shape of the pupil and the color characteristics of the dark pixels, so that the pupil coordinates are determined.
In one embodiment of the present disclosure, cropping a video including left and right eye portions in a face video with pupil coordinates as a center to obtain a first eye video includes:
and cutting each first frame image by taking pupil coordinates as a center to obtain a plurality of left and right eye images, and converting the left and right eye images into videos to obtain a first eye video.
In this embodiment, after determining the coordinates of the pupil in the first frame image, a square image with a size of 384×384 pixels is cut out by using the pupil coordinates as the center point of the region, and the same operation is performed on each first frame image to obtain a plurality of left and right eye images. These cropped left and right eye images can be converted into video by the ffmpeg video component, resulting in a first eye video.
In one embodiment of the present disclosure, extracting an eye movement feature in a second eye video to obtain feature pictures of lateral movement of a plurality of eyeballs over time includes:
and obtaining a plurality of continuous second frame images from the second eye video extraction frame, determining pupil coordinates of eyeballs in the second frame images and horizontal line segments with preset lengths taking the pupil coordinates as centers, intercepting rectangular areas which are wide by the horizontal line segments and long by line segments perpendicular to the horizontal line segment directions in the second frame images to obtain a plurality of third frame images, and splicing the plurality of third frame images in time sequence to obtain characteristic pictures of the eyeballs moving transversely along with time.
In this embodiment, the second eye video contains abundant eye movement features, and the most abundant eye movement features are the most common lateral line of sight changes in the reading process, and the extraction steps are as follows:
And extracting frames from the second eye video to obtain a plurality of continuous second frame images, determining pupil coordinates of eyeballs in the second frame images and a horizontal line segment with a preset length taking the pupil as a center, intercepting a rectangular area which is wide by the horizontal line segment and long by the line segment perpendicular to the direction of the horizontal line segment in the second frame images to obtain a plurality of third frame images, and splicing the plurality of third frame images in time sequence to obtain a characteristic picture of the eyeballs moving transversely along with time.
In one embodiment of the present disclosure, referring to fig. 2, before inputting the first eye video into the pre-training amplification network to obtain the second eye video, further comprising:
inputting the third eye video and the fourth eye video into a pre-training amplifying network to obtain a sixth eye video, determining error loss of the sixth eye video and a preset fifth eye video, and correcting the weight coefficient of the pre-training amplifying network based on the error loss to obtain a fine-tuned pre-training amplifying network;
The pre-training amplifying network is an amplifying network based on micro motion of multiple types of objects.
In this embodiment, the pretraining amplification network for amplifying micro-motion of multiple types of objects is trained on a wide synthetic data set, and learns micro-motion amplification processes (such as tables and chairs, vegetation, automobiles, animals, etc.) of objects commonly found in nature. The embodiment is based on the pretraining amplifying network for amplifying the micro motion of various objects, and after the pretraining amplifying network is finely tuned on a specific eye micro motion image, the amplifying function of the pretraining network is smoothly migrated to a human eye micro motion scene.
Specific eye micro-motion images comprise an original human eye image, a micro-motion human eye image and a micro-motion human eye amplified image. The human eye original image is used for determining a human eye original fixation point; the human eye micro-motion image is used for simulating the state of human eye micro-motion, and eyeballs, sclera, eyelashes, periocular muscles and the like also follow to do micro-motion; the human eye micro-amplification image is used for multiplying the gazing displacement of the micro-amplification image relative to the original image by the amplification factor to obtain the micro-amplified human eye image.
The third eye video is a video obtained by restoring all original images of human eyes, the fourth eye video is a video obtained by restoring all micro-motion images of human eyes, the fifth eye video is a video obtained by restoring all micro-motion amplification images of human eyes, and the sixth eye video is a video obtained by inputting the third eye video and the fourth eye video into a pre-training amplification network.
A particular eye micro-image may be generated using UnityEyes render 3D human eye image tool: an Interactive interaction mode in the tool is opened, so that an eyeball moves along with the position of a mouse, the mouse moves from an original position to a jogging position and then to an amplifying position, and a Frame A (human eye original image), a Frame B (human eye jogging image) and a Frame M (human eye jogging amplifying image) are respectively generated, so that the effects of gazing at different positions and gazing displacement amplifying are achieved. A total of 528,138 pictures, i.e. 176,046 sets of frameA, frameB and frameM pixels of size 384 x 384, are generated. The fine tuning process of the pre-training amplifying network is as follows: the shallow layer features and the deep layer features of the inputted frames A and B are extracted by using a pre-training amplifying network and then are connected through residual errors, and the difference value of the features of the shallow layer features and the deep layer features is amplified by an amplifying coefficientAfter amplification, the original frames are superimposed to obtain a Frame M' and a generated Frame M as MSE mean square error loss, and the weight coefficient of the pre-training amplification network is corrected based on the error loss to obtain a fine-tuned pre-training amplification network.
The amplification effect of a fine-tuned pre-trained amplification network can be expressed in terms of index structural similarity (Structural Similarity Index, SSIM) and peak signal-to-Noise Ratio (PSNR).
SSIM: the similarity of structure, brightness and contrast of the two videos is measured. An SSIM value of 1 indicates perfect replication, and a value below 1 indicates a difference.
PSNR: similar to SNR but more commonly used for image and video quality assessment. A high PSNR generally means a higher quality.
SSIM and PSNR are average values obtained by comparing each frame of video with each frame of original video, referring to table 1.
TABLE 1
In one embodiment of the present disclosure, referring to fig. 4, a personality classification network is configured to extract a plurality of features in a feature picture, and classify the plurality of features to obtain a personality evaluation result;
The plurality of features includes: blink times, blink time intervals, number of hops, time intervals, distance between hops, pupil diameter variation; the characteristic picture is a scanning line graph;
Wherein, extracting the blink number comprises:
detecting the number of lines in a scanning line graph, and determining the number of the lines as blink times;
Extracting the blink interval includes:
Detecting a first horizontal distance between lines in a scanning line diagram, and determining the first horizontal distance as a blink time interval;
extracting the number of hops includes:
Determining the position of a pupil on a horizontal line segment in a scanning line graph, detecting the number of black strips with the position of the pupil suddenly and vertically displaced in the vertical direction, and determining the number of the black strips as the number of eye hops;
extracting the eye-hop time interval includes:
detecting a second horizontal distance between black stripes of abrupt vertical displacement in the scan line graph, and determining the second horizontal distance as an eye jump time interval;
Extracting the eye jump distance includes:
Detecting the distance of the black stripe moving along the vertical direction, and determining the distance as the eye jump distance;
Extracting pupil diameter variations includes:
The length of the black band at different times was detected and determined as the pupil diameter.
In this embodiment, a plurality of features in the feature image are extracted, the plurality of features are formed into feature vectors, and then the feature vectors are input into a personality classification network, and the personality classification network classifies the feature vectors to obtain a personality evaluation result. The plurality of features includes: blink times, blink time intervals, number of hops, time intervals, distance between hops, pupil diameter variation; the feature pictures are scan line graphs.
Scan line patterns, also known as space-time or oscillograms, may represent the change in pixel intensity over time over a segment taken in a video. Eye movement changes in the video are tracked by this visualization to analyze eye movements such as blinks and saccades. The vertical axis of fig. 4 corresponds to all pixels on the segment taken, and the horizontal axis of the picture corresponds to time, i.e., frame variation.
The feature pictures comprise rich time sequence data of human eyes when a target user reads: blink times, blink time intervals, number of hops, hop time intervals, hop distance, pupil diameter variation, etc.
Blinking: the blinks are shown as vertical bands in fig. 4, with intensity varying across the width of the image, indicating the duration of eyelid closure.
Number of blinks: the number of these vertical strips was counted. The pixel intensity variation is analyzed by using an edge detection algorithm (e.g., sobel operator) and then using Hough transform to detect the line. After detecting the lines, counting the number of the lines.
Blink interval: the horizontal distance between these vertical bands is statistically measured to determine the time interval for each blink. The horizontal distance between the above-mentioned vertical lines is measured in units of pixels, and corresponds to the number of frames between the events represented. The relationship between the number of frames Fblink between blink time interval Tblink and blink event and frame rate FRAME RATE may be expressed as:
Tblink = Fblink/FRAME RATE (where the frame rate FRAME RATE is typically 30 fps)
Eye jump (saccade): representing a vertical displacement of the position of the pupil or iris pattern along the line segment.
Number of eye hops: the position of the pupil on the taken line segment is located by the position of the black pixel. The number of black bands that suddenly shift vertically is identified and counted.
Eye jump time interval: the horizontal distance between the black stripes of abrupt vertical displacement is counted to determine the time interval between each eye jump. The relationship between the frame number Fsaccade between the eye-skip time interval Tsaccade and the eye-skip event and the frame rate FRAME RATE can be expressed as:
Tsaccade = Fsaccade/FRAME RATE (where the frame rate FRAME RATE is typically 30 fps)
Eye jump distance: distance between two hops the black stripe is displaced in the vertical direction.
Pupil diameter variation: the pupil diameter change is manifested as a change in the length of the dark region over time by the position of the black pixel locating the pupil on the taken line segment. Pupil diameters at different time points can be calculated by measuring the lengths of the black stripes at different time points.
Corresponding to the personality evaluation method of the above embodiment, fig. 5 is a block diagram of a personality evaluation system according to an embodiment of the present disclosure. For ease of illustration, only portions relevant to embodiments of the present disclosure are shown. Referring to fig. 5, the personality assessment system 20 includes: the eye motion amplifying module 21, the characteristic picture obtaining module 22 and the personality evaluation result obtaining module 23.
The eye motion amplifying module 21 is configured to input the first eye video into the pre-training amplifying network to amplify the eye motion, so as to obtain a second eye video; the first eye video is an eye motion video when a target user reads a material;
The feature image obtaining module 22 is configured to extract an eye movement feature in the second eye video, so as to obtain feature images of lateral movement of multiple eyeballs along with time;
and the personality evaluation result acquisition module 23 is configured to input the feature picture into a personality classification network to obtain a personality evaluation result of the target user.
In one embodiment of the present disclosure, the eye motion amplification module 21 is further configured to:
Determining pupil coordinates in the facial video;
And cutting out the video containing the left eye part and the right eye part in the face video by taking the pupil coordinates as the center to obtain a first eye video.
In one embodiment of the present disclosure, the eye motion amplification module 21 is specifically configured to:
And extracting frames from the facial video to obtain a plurality of continuous first frame images, determining coordinates corresponding to left and right eyes in each first frame image, and determining pupil coordinates according to the characteristics of pupils and the coordinates corresponding to the left and right eyes.
In one embodiment of the present disclosure, the eye motion amplification module 21 is specifically configured to:
And cutting each first frame image by taking the pupil coordinates as the center to obtain a plurality of left and right eye images, and converting the left and right eye images into videos to obtain a first eye video.
In one embodiment of the present disclosure, the feature picture acquisition module 22 is specifically configured to:
Obtaining a plurality of continuous second frame images from a second eye video extraction frame, and determining pupil coordinates of eyeballs in the second frame images and horizontal line segments with preset lengths taking the pupil coordinates as centers;
and intercepting a rectangular area which is wide by a horizontal line segment and long by a line segment vertical to the direction of the horizontal line segment in the second frame image to obtain a plurality of third frame images, and splicing the plurality of third frame images according to time sequence to obtain a characteristic picture of the eyeball moving transversely along with time.
In one embodiment of the present disclosure, the eye motion amplification module 21 is specifically configured to:
inputting the third eye video and the fourth eye video into a pre-training amplifying network to obtain a sixth eye video, determining error loss of the sixth eye video and a preset fifth eye video, and correcting the weight coefficient of the pre-training amplifying network based on the error loss to obtain a fine-tuned pre-training amplifying network;
The pre-training amplifying network is an amplifying network based on micro motion of multiple types of objects.
In one embodiment of the disclosure, the personality classification network is configured to extract a plurality of features in the feature image, and classify the plurality of features to obtain a personality evaluation result;
The plurality of features includes: blink times, blink time intervals, number of hops, time intervals, distance between hops, pupil diameter variation; the characteristic picture is a scanning line graph;
Wherein, extracting the blink number comprises:
detecting the number of lines in the scanning line diagram, and determining the number of lines as blink times;
Extracting the blink interval includes:
Detecting a first horizontal distance between lines in the scanning line diagram, and determining the first horizontal distance as a blink time interval;
extracting the number of hops includes:
determining the position of a pupil on the horizontal line segment in the scanning line graph, detecting the number of black strips with the position of the pupil generating sudden vertical displacement in the vertical direction, and determining the number of the black strips as the number of hops;
extracting the eye-hop time interval includes:
detecting a second horizontal distance between black stripes of abrupt vertical displacement in the scan line graph, and determining the second horizontal distance as an eye jump time interval;
Extracting the eye jump distance includes:
detecting a distance of the black stripe moving in a vertical direction, and determining the distance as an eye jump distance;
Extracting pupil diameter variations includes:
the length of the black stripe is detected at different times, and the length of the black stripe is determined as the pupil diameter.
Referring to fig. 6, fig. 6 is a schematic block diagram of an electronic device according to an embodiment of the present disclosure. The electronic device 300 in the present embodiment as shown in fig. 6 may include: one or more processors 301, one or more input devices 302, one or more output devices 303, and one or more memories 304. The processor 301, the input device 302, the output device 303, and the memory 304 communicate with each other via a communication bus 305. The memory 304 is used to store a computer program comprising program instructions. The processor 301 is configured to execute program instructions stored in the memory 304. Wherein the processor 301 is configured to invoke program instructions to perform the functions of the modules/units of the system embodiments described above, such as the functions of the modules 21 to 23 shown in fig. 5.
It should be appreciated that in the disclosed embodiments, the Processor 301 may be a central processing unit (Central Processing Unit, CPU), which may also be other general purpose processors, digital signal processors (DIGITAL SIGNAL processors, DSPs), application SPECIFIC INTEGRATED Circuits (ASICs), off-the-shelf Programmable gate arrays (Field-Programmable GATE ARRAY, FPGA) or other Programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
The input device 302 may include an image pickup device, a touch pad, a fingerprint sensor (for collecting fingerprint information of a user and direction information of a fingerprint), a microphone, etc., and the output device 303 may include a display (LCD, etc.), a speaker, etc.
The memory 304 may include read only memory and random access memory and provides instructions and data to the processor 301. A portion of memory 304 may also include non-volatile random access memory. For example, the memory 304 may also store information of device type.
In a specific implementation, the processor 301, the input device 302, and the output device 303 described in the embodiments of the present disclosure may perform the implementation described in the first embodiment and the second embodiment of the personality evaluation method provided in the embodiments of the present disclosure, and may also perform the implementation of the electronic device described in the embodiments of the present disclosure, which is not described herein again.
In another embodiment of the disclosure, a computer readable storage medium is provided, where the computer readable storage medium stores a computer program, where the computer program includes program instructions, where the program instructions, when executed by a processor, implement all or part of the procedures in the method embodiments described above, or may be implemented by instructing related hardware by the computer program, where the computer program may be stored in a computer readable storage medium, where the computer program, when executed by the processor, implements the steps of each of the method embodiments described above. Wherein the computer program comprises computer program code, which may be in the form of source code, object code, executable files or in some intermediate form, etc. The computer readable medium may include: any entity or device capable of carrying computer program code, a recording medium, a U disk, a removable hard disk, a magnetic disk, an optical disk, a computer Memory, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), an electrical carrier signal, a telecommunications signal, a software distribution medium, and so forth. It should be noted that the content of the computer readable medium can be appropriately increased or decreased according to the requirements of the jurisdiction's jurisdiction and the patent practice, for example, in some jurisdictions, the computer readable medium does not include electrical carrier signals and telecommunication signals according to the jurisdiction and the patent practice.
The computer readable storage medium may be an internal storage unit of the electronic device of any of the foregoing embodiments, such as a hard disk or a memory of the electronic device. The computer readable storage medium may also be an external storage device of the electronic device, such as a plug-in hard disk provided on the electronic device, a smart memory card (SMART MEDIA CARD, SMC), a Secure Digital (SD) card, a flash memory card (FLASH CARD), or the like. Further, the computer-readable storage medium may also include both internal storage units and external storage devices of the electronic device. The computer-readable storage medium is used to store a computer program and other programs and data required for the electronic device. The computer-readable storage medium may also be used to temporarily store data that has been output or is to be output.
Those of ordinary skill in the art will appreciate that the elements and algorithm steps described in connection with the embodiments disclosed herein may be embodied in electronic hardware, in computer software, or in a combination of the two, and that the elements and steps of the examples have been generally described in terms of function in the foregoing description to clearly illustrate the interchangeability of hardware and software. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present disclosure.
It will be clearly understood by those skilled in the art that, for convenience and brevity of description, the specific working process of the electronic device and unit described above may refer to the corresponding process in the foregoing method embodiment, which is not repeated herein.
In the several embodiments provided in the present application, it should be understood that the disclosed electronic device and method may be implemented in other manners. For example, the apparatus embodiments described above are merely illustrative, e.g., the division of elements is merely a logical functional division, and there may be additional divisions of actual implementation, e.g., multiple elements or components may be combined or integrated into another system, or some features may be omitted, or not performed. In addition, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection via some interfaces or units, or may be an electrical, mechanical, or other form of connection.
The units described as separate units may or may not be physically separate, and units shown as units may or may not be physical units, may be located in one place, or may be distributed over a plurality of network units. Some or all of the units may be selected according to actual needs to achieve the purposes of the embodiments of the present disclosure.
In addition, each functional unit in each embodiment of the present disclosure may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit. The integrated units may be implemented in hardware or in software functional units.
The foregoing is merely a specific embodiment of the present disclosure, but the protection scope of the present disclosure is not limited thereto, and any equivalent modifications or substitutions will be apparent to those skilled in the art within the scope of the present disclosure, and these modifications or substitutions should be covered in the scope of the present disclosure. Therefore, the protection scope of the present disclosure shall be subject to the protection scope of the claims.
Claims (10)
1. A personality assessment method comprising:
Inputting the first eye video into a pre-training amplifying network to amplify eye actions, so as to obtain a second eye video; the first eye video is an eye motion video when a target user reads a material;
Extracting eye movement characteristics in the second eye video to obtain characteristic pictures of lateral movement of a plurality of eyeballs along with time;
and inputting the characteristic pictures into a personality classification network to obtain a personality evaluation result of the target user.
2. The personality assessment method of claim 1, further comprising:
Determining pupil coordinates in the facial video;
And cutting out the video containing the left eye part and the right eye part in the face video by taking the pupil coordinates as the center to obtain a first eye video.
3. The personality assessment method of claim 2, wherein the determining pupil coordinates in the facial video includes:
And extracting frames from the facial video to obtain a plurality of continuous first frame images, determining coordinates corresponding to left and right eyes in each first frame image, and determining pupil coordinates according to the characteristics of pupils and the coordinates corresponding to the left and right eyes.
4. The personality assessment method according to claim 2, wherein the cropping out the video including the left and right eye portions from the face video with the pupil coordinates as a center to obtain the first eye video includes:
And cutting each first frame image by taking the pupil coordinates as the center to obtain a plurality of left and right eye images, and converting the left and right eye images into videos to obtain a first eye video.
5. The personality assessment method according to claim 1, wherein the extracting the eye movement features in the second eye video to obtain feature pictures of lateral movement of a plurality of eyeballs over time includes:
Obtaining a plurality of continuous second frame images from a second eye video extraction frame, and determining pupil coordinates of eyeballs in the second frame images and horizontal line segments with preset lengths taking the pupil coordinates as centers;
and intercepting a rectangular area which is wide by a horizontal line segment and long by a line segment vertical to the direction of the horizontal line segment in the second frame image to obtain a plurality of third frame images, and splicing the plurality of third frame images according to time sequence to obtain a characteristic picture of the eyeball moving transversely along with time.
6. The personality assessment method of claim 1, wherein before inputting the first eye video into the pre-training amplification network to obtain the second eye video, further comprising:
Inputting a third eye video and a fourth eye video into the pre-training amplifying network to obtain a sixth eye video, determining error loss of the sixth eye video and a preset fifth eye video, and correcting the weight coefficient of the pre-training amplifying network based on the error loss to obtain a fine-tuned pre-training amplifying network;
The pre-training amplifying network is an amplifying network based on micro motion of multiple types of objects.
7. The personality assessment method according to claim 1, wherein the personality classification network is configured to extract a plurality of features in the feature picture, and classify the plurality of features to obtain a personality assessment result;
The plurality of features includes: blink times, blink time intervals, number of hops, time intervals, distance between hops, pupil diameter variation; the characteristic picture is a scanning line graph;
Wherein, extracting the blink number comprises:
detecting the number of lines in the scanning line diagram, and determining the number of lines as blink times;
Extracting the blink interval includes:
Detecting a first horizontal distance between lines in the scanning line diagram, and determining the first horizontal distance as a blink time interval;
extracting the number of hops includes:
determining the position of a pupil on the horizontal line segment in the scanning line graph, detecting the number of black strips with the position of the pupil generating sudden vertical displacement in the vertical direction, and determining the number of the black strips as the number of hops;
extracting the eye-hop time interval includes:
detecting a second horizontal distance between black stripes of abrupt vertical displacement in the scan line graph, and determining the second horizontal distance as an eye jump time interval;
Extracting the eye jump distance includes:
detecting a distance of the black stripe moving in a vertical direction, and determining the distance as an eye jump distance;
Extracting pupil diameter variations includes:
the length of the black stripe is detected at different times, and the length of the black stripe is determined as the pupil diameter.
8. A personality assessment system, comprising:
The eye motion amplifying module is used for inputting the first eye video into the pre-training amplifying network to amplify eye motion so as to obtain a second eye video; the first eye video is an eye motion video when a target user reads a material;
The characteristic picture acquisition module is used for extracting eye movement characteristics in the second eye video to obtain characteristic pictures of lateral movement of a plurality of eyeballs along with time;
And the personality evaluation result acquisition module is used for inputting the characteristic pictures into a personality classification network to obtain the personality evaluation result of the target user.
9. An electronic device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that the processor implements the steps of the method according to any one of claims 1 to 7 when the computer program is executed.
10. A computer readable storage medium storing a computer program, characterized in that the computer program when executed by a processor implements the steps of the method according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410215888.4A CN118038164A (en) | 2024-02-27 | 2024-02-27 | Personality assessment method and system for eye movement video amplification based on camera device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410215888.4A CN118038164A (en) | 2024-02-27 | 2024-02-27 | Personality assessment method and system for eye movement video amplification based on camera device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN118038164A true CN118038164A (en) | 2024-05-14 |
Family
ID=90990955
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202410215888.4A Pending CN118038164A (en) | 2024-02-27 | 2024-02-27 | Personality assessment method and system for eye movement video amplification based on camera device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN118038164A (en) |
-
2024
- 2024-02-27 CN CN202410215888.4A patent/CN118038164A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110623629B (en) | Visual attention detection method and system based on eyeball motion | |
| US10188338B2 (en) | Content evaluation system and content evaluation method using the system | |
| Itti | Quantifying the contribution of low-level saliency to human eye movements in dynamic scenes | |
| EP2829221B1 (en) | Asperger's diagnosis assistance device | |
| Gredebäck et al. | Eye tracking in infancy research | |
| US20170027805A1 (en) | Vision-Based Diagnosis and Treatment | |
| US9538947B2 (en) | Method, system and device for assisting diagnosis of autism | |
| CN111933275B (en) | Depression evaluation system based on eye movement and facial expression | |
| US11928632B2 (en) | Ocular system for deception detection | |
| Sivasangari et al. | Emotion recognition system for autism disordered people | |
| KR20200012355A (en) | Online lecture monitoring method using constrained local model and Gabor wavelets-based face verification process | |
| Hahn et al. | Thatcherization impacts the processing of own-race faces more so than other-race faces: An ERP study | |
| Meinhardt-Injac et al. | The time course of face matching by internal and external features: Effects of context and inversion | |
| CN115334957A (en) | System and method for optical assessment of pupillary psychosensory response | |
| US9355366B1 (en) | Automated systems for improving communication at the human-machine interface | |
| CN114129165A (en) | Psychological assessment method, system and storage medium based on credible assessment scale | |
| Amudha et al. | A fuzzy based eye gaze point estimation approach to study the task behavior in autism spectrum disorder | |
| CN118038164A (en) | Personality assessment method and system for eye movement video amplification based on camera device | |
| CN108495584B (en) | Apparatus and method for determining eye movement through a haptic interface | |
| CN115299945A (en) | Attention and fatigue degree evaluation method and wearable device | |
| Stöckli et al. | A practical guide for automated facial emotion classification 1 | |
| Bennett et al. | Looking at faces: autonomous perspective invariant facial gaze analysis | |
| KR102625583B1 (en) | Virtual reality based visual perseption analysis system, virtual reality based visual perseption analysis method and computer program stored in a recording medium to execute the method thereof | |
| Dollée | The Role of the Eyes in the Uncanny Valley Effect: Does Incongruence between Eyes and Face Influence Uncanniness? | |
| KR102616230B1 (en) | Method for determining user's concentration based on user's image and operating server performing the same |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |