CN112674770A - Depression crowd eye movement identification method based on image significance difference and emotion analysis - Google Patents
Depression crowd eye movement identification method based on image significance difference and emotion analysis Download PDFInfo
- Publication number
- CN112674770A CN112674770A CN202011530259.9A CN202011530259A CN112674770A CN 112674770 A CN112674770 A CN 112674770A CN 202011530259 A CN202011530259 A CN 202011530259A CN 112674770 A CN112674770 A CN 112674770A
- Authority
- CN
- China
- Prior art keywords
- emotion
- difference
- images
- tester
- image
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Abstract
The application discloses a method for identifying eye movement of depressed people based on image significance difference and emotion analysis, wherein the method comprises the following steps: acquiring eye movement track data of a tester in a first preset time; according to the significance difference detection network and the eye movement track data, obtaining significance fixation difference graphs of normal people and depressed people; generating a distribution characteristic diagram of the emotion image in each emotion dimension through a preset weak supervision coupling network; and obtaining the difference of the attention mechanism of the normal population and the depression population according to the distribution characteristic diagram of the significant gazing difference and the emotion dimensionality so as to identify the depression state according to the difference of the attention mechanism. Therefore, the eye movement track data of the testee in the experimental process is analyzed from two aspects of the attention mechanism, the difference between normal people and depression patients is mined, corresponding features are extracted to represent the psychological state of the testee, and therefore the task of identifying the depression state is completed quickly and accurately.
Description
Technical Field
The application relates to the technical field of visual analysis, in particular to a method for identifying eye movements of depressed people based on image significance difference and emotion analysis.
Background
As society develops, people's physical lives are greatly enriched, but the pressure from work and life makes more and more people suffer from mental diseases. Depression is a common mental disease, and is different from common mood fluctuation, patients are easy to have sadness, guilt, self-negation, lack of interest in things, and have the phenomena of insomnia, fatigue, inappetence, inattention and the like, so that the work and the life of the patients are greatly influenced. In severe cases, depression can lead to suicide, more than 3 million people worldwide suffer from depression but less than half of the patients receive effective treatment, nearly 80 million people worldwide die of suicide due to depression every year, and the disease is one of the most serious mental diseases at present.
There are various factors that affect the effective treatment of patients with depression, including lack of adequate medical resources, lack of trained psychologists, general social discrimination against mental diseases, etc., but the most important causes are the inability to accurately and effectively identify depression. Compared with physiological diseases, the mental disease detection load is heavier, the detection period is longer, and the detection precision is lower. Therefore, how to accurately and effectively identify depression is always a hot topic in the fields of psychology and signal processing.
The existing depression state identification method mainly depends on clinical diagnosis and scale evaluation, so that a large amount of medical resources are consumed, and subjective judgment exists at the same time, so that the problem is urgently solved.
Content of application
The method for identifying the eye movement of the depressed people based on the significant difference and emotion analysis of the images analyzes the eye movement track data of a tester in the experimental process from two aspects of an attention mechanism, excavates the difference between normal people and depression patients, and extracts corresponding characteristics to represent the psychological state of the tester, thereby quickly and accurately completing the task of identifying the depressed state.
The embodiment of the first aspect of the application provides a method for identifying the eye movement of a depressed person based on the significant difference and emotion analysis of images, which comprises the following steps:
acquiring eye movement track data of a tester in a first preset time;
according to the significance difference detection network and the eye movement track data, obtaining significance fixation difference graphs of normal people and depressed people;
generating a distribution characteristic diagram of the emotion images in each emotion dimension through a preset weak supervision coupling network and the emotion images; and
and obtaining the difference of the attention mechanism of the normal population and the depressed population according to the significant gazing difference and the distribution characteristic diagram of the emotional dimension, so as to identify the depressed state according to the difference of the attention mechanism. Optionally, the acquiring eye movement trajectory data of the tester within a first preset time includes:
presenting a black background image on a screen, and observing the screen by the tester within a second preset time to release the current attention of the tester;
a white cross appears in the right center of the screen, and the tester concentrates the sight on the right center of the screen for a third preset time;
the screen presents a first group of positive emotion images and negative emotion images and lasts for a fourth preset time;
the tester observes the first group of positive emotion images and any area of the negative emotion images;
and returning to execute the step of presenting a black background image on the screen, and observing the screen by the tester in a second preset time to relieve the current attention of the tester until the positive emotional images and the negative emotional images presented on the screen reach a preset group number.
Optionally, the obtaining a significant fixation difference map of a normal population and a depressed population according to the significant difference detection network and the eye movement trajectory data includes:
obtaining a watching distribution diagram of the normal person for the emotional image through the significance detection network of the upper side branch and the eye movement track data;
obtaining a watching distribution diagram of the depressed people for the emotional image through a significance detection network of a lower branch and the eye movement track data;
and splicing 64-channel significance characteristic graphs generated by the convolution operation of the upper branch and the lower branch by 3 x 3 to obtain a 128-channel significance characteristic graph, and obtaining a significance fixation difference distribution map of the normal population and the depressed population according to the 128-channel significance characteristic graph and the eye movement track data.
Optionally, the generating a distribution feature map of the emotion image in each emotion dimension through a preset weak supervision coupling network and the emotion image includes:
acquiring deep network characteristics corresponding to the positive emotion image and the negative emotion image;
determining an emotion distribution prediction branch according to the deep network characteristics;
and generating a distribution characteristic diagram of the emotion image in each emotion dimension according to the emotion distribution prediction branch and the over-preset weak supervision coupling network.
Optionally, the emotion dimensions include:
eight emotional dimensions of entertainment, anger, feast, satisfaction, disgust, excitement, fear, and sadness.
In a second aspect, an embodiment of the present application provides a device for identifying eye movements of depressed people based on significant difference and mood analysis of images, including:
the acquisition module is used for acquiring eye movement track data of a tester in first preset time;
the acquisition module is used for detecting a network and the eye movement track data according to the significance difference to obtain a significance gazing difference diagram of normal people and depressed people;
the generating module is used for generating a distribution characteristic diagram of the emotion images in each emotion dimension through a preset weak supervision coupling network and the emotion images; and
and the identification module is used for obtaining the difference of attention mechanism between the normal population and the depressed population according to the significant gazing difference and the distribution characteristic diagram of the emotion dimension so as to identify the depression state according to the difference of attention mechanism. Optionally, the acquisition module is specifically configured to:
presenting a black background image on a screen, and observing the screen by the tester within a second preset time to release the current attention of the tester;
a white cross appears in the right center of the screen, and the tester concentrates the sight on the right center of the screen for a third preset time;
the screen presents a first group of positive emotion images and negative emotion images and lasts for a fourth preset time;
the tester observes the first group of positive emotion images and any area of the negative emotion images;
and returning to execute the step of presenting a black background image on the screen, and observing the screen by the tester in a second preset time to relieve the current attention of the tester until the positive emotional images and the negative emotional images presented on the screen reach a preset group number.
Optionally, the obtaining module is specifically configured to:
obtaining a watching distribution diagram of the normal person for the emotional image through the significance detection network of the upper side branch and the eye movement track data;
obtaining a watching distribution diagram of the depressed people for the emotional image through a significance detection network of a lower branch and the eye movement track data;
and splicing 64-channel significance characteristic graphs generated by the convolution operation of the upper branch and the lower branch by 3 x 3 to obtain a 128-channel significance characteristic graph, and obtaining a significance fixation difference distribution map of the normal population and the depressed population according to the 128-channel significance characteristic graph and the eye movement track data.
Optionally, the generating module is specifically configured to:
acquiring deep network characteristics corresponding to the positive emotion image and the negative emotion image;
determining an emotion distribution prediction branch according to the deep network characteristics;
and generating a distribution characteristic diagram of the emotion image in each emotion dimension according to the emotion distribution prediction branch and the over-preset weak supervision coupling network.
Optionally, the emotion dimensions include:
eight emotional dimensions of entertainment, anger, feast, satisfaction, disgust, excitement, fear, and sadness.
The method comprises the steps of constructing eye movement track data sets for normal people and depression patients by designing a visual significance experimental paradigm combined with image semantic ideas, analyzing the difference of two populations on an attention mechanism by extracting significance difference characteristics and emotion distribution characteristics of experimental images, and further realizing classification tasks for the two populations. The free-viewing experimental paradigm adopts a combination of a positive emotion image and a negative emotion image as visual stimuli, a tester is required to freely view in an image area, and the whole process simulates a behavior pattern of the tester when the tester freely views in daily life. Additional aspects and advantages of the present application will be set forth in part in the description which follows and, in part, will be obvious from the description, or may be learned by practice of the present application.
Drawings
The foregoing and/or additional aspects and advantages of the present application will become apparent and readily appreciated from the following description of the embodiments, taken in conjunction with the accompanying drawings of which:
fig. 1 is a flowchart of a method for identifying eye movements of a depressed person based on significant differences in images and mood analysis according to an embodiment of the present application;
FIG. 2 is a data collection flow of an experimental paradigm of visual saliency according to one embodiment of the present application;
FIG. 3 is a single test flow of an experimental paradigm of visual saliency according to one embodiment of the present application;
FIG. 4 is a schematic view of fixation profiles corresponding to emotion images according to an embodiment of the present application;
FIG. 5 is a schematic diagram of a three-branch significance difference detection network, according to one embodiment of the present application;
FIG. 6 is a schematic diagram of a weakly supervised coupling network for image emotion analysis according to an embodiment of the present application;
FIG. 7 is a schematic diagram of a visual attention-based depressive state recognition study work framework according to one embodiment of the present application;
fig. 8 is a block schematic diagram of a device for identifying eye movements of depressed people based on significant difference of images and mood analysis according to an embodiment of the present application.
Detailed Description
Reference will now be made in detail to embodiments of the present application, examples of which are illustrated in the accompanying drawings, wherein like or similar reference numerals refer to the same or similar elements or elements having the same or similar function throughout. The embodiments described below with reference to the drawings are exemplary and intended to be used for explaining the present application and should not be construed as limiting the present application.
The method for identifying the eye movement of a depressed person based on the analysis of the significant difference and the mood of the image according to the embodiment of the present application is described below with reference to the drawings.
Before describing the method for identifying the eye movement of a depressed person based on the analysis of the significant difference and the mood of the image according to the embodiment of the present application, a method for identifying a depressed state in the related art will be briefly described.
In the related art, there are mainly the following two depression state identification methods: clinical diagnostic methods and structured metrology methods. Specifically, the clinician analyzes the degree of depressive state of the test subject by analyzing the oral depression indicators of the test subject, including monotonous tone, decreased speech rate, decreased volume, etc., and the non-oral depression indicators of the test subject, including reduced gestures, increased downward sightline, etc.; the structured quantitative table analyzes the degree of the depression state of the testers from multiple dimensions, and integrates the scoring conditions of all the dimensions to obtain a final depression state evaluation result.
However, the two methods not only consume a large amount of medical resources, but also have subjective judgment.
Therefore, the present application is based on the above-mentioned drawbacks, and proposes a method for identifying the eye movements of depressed people based on the significant difference and mood analysis of images.
Specifically, fig. 1 is a schematic flowchart of an eye movement identification method for a depressed person based on significant difference and mood analysis of images according to an embodiment of the present application.
As shown in fig. 1, the method for identifying the eye movement of the depressed people based on the significant difference and emotion analysis of the images comprises the following steps:
in step S101, eye movement trajectory data of the tester within a first preset time is collected. The test subjects included, among others, the normal population and the depressed population.
It will be appreciated that a number of psychological studies have shown that the factors responsible for depression are multifaceted, with cognitive factors, particularly attention bias (i.e. processing bias towards negative stimuli), being one of the important causes for the development, maintenance and progression of depression. It is noted that the first step of the cognitive process, which is also the primary link in the processing of external stimuli, is that some stimuli are processed and amplified preferentially and others are ignored due to the presence of a preference for attention. Patients with depression often exhibit selective attention to negative stimuli and maintain this poor information processing modality, resulting in a poor mental state.
In the real world, people often understand the surrounding environment in a free-viewing manner, and the attention mode of each person to the external environment is different in combination with the living habits and the growth experiences of the people. For example, in the study of autism, it is widely believed that there are distinct patterns of attention between normal people and patients with autism, such as normal people tend to look over the faces of different people and guess their interrelationships in one image, while patients with depression tend to focus on someone or an object in the center of the image. Similarly, in the study of depression, the theory of attention bias further proves that different attention modes exist between normal people and depression patients, in order to explore the difference of the attention modes of the two groups, a set of visual significance experimental paradigm combining image semantic ideas is designed, and the attention mechanism of the two groups in the free watching mode is analyzed.
Optionally, in some embodiments, the collecting the eye movement trajectory data of the tester within the first preset time includes: presenting a black background image on the screen, and enabling a tester to observe the screen within a second preset time so as to relieve the current attention of the tester; a white cross is displayed in the right center of the screen, and the tester concentrates the sight on the right center of the screen for a third preset time; presenting a first group of positive emotion images and negative emotion images on a screen, and continuing for a fourth preset time; observing any area of the first group of positive emotion images and the negative emotion images by a tester; and returning to execute the step of presenting a black background image on the screen, and observing the screen by the tester within a second preset time to relieve the current attention of the tester until the positive emotion images and the negative emotion images presented on the screen reach a preset group number.
Specifically, in the experiment, in conjunction with fig. 2 and 3, the tester was asked to sit flat at a position 60cm from the evaluation apparatus and to start the experiment by itself in accordance with the instruction of the experiment. Firstly, a black background image is presented to a tester on a screen, and the purpose is to relieve the current attention of the tester and enable the tester to randomly watch any area on the screen; when the second preset time, t1At any moment, a white cross appears in the center of the screen, so that a tester concentrates the sight on the center of the screen; when the third preset time, t2Screening a positive emotion image and a negative emotion image of the same category according to the joyfulness, and simultaneously displaying the positive emotion image and the negative emotion image on a screen; when the fourth preset time is reached, i.e.t3At that time, the tester can freely view any area in the two images, and the test process is finished.
The whole process includes 80 times of the above experimental procedures, wherein 20 groups of emotion image combinations of human category, 20 groups of emotion image combinations of animal category, 20 groups of emotion image combinations of object category, and 20 groups of emotion image combinations of scene category, and a total of 160 emotion images. During the whole experimental test process, 80 groups of visual stimuli appear in a random and disorganized sequence, and the left and right positions of the two emotion images also appear randomly, so that a tester is required to keep attention all the time during the experimental process to browse the two emotion images and generate more attentions to the interested area.
That is, the application can record t from the tester through the eye movement track capture module of the evaluation device2Time t3All eye movement trajectories within the two emotion image regions at the time are used to generate their attention profiles for the two emotion images. Each tester performs 80 groups of experiments in the experiment process, the total experiment time is about 7 minutes, namely, the embodiment of the application can simultaneously present a pair of emotion images (positive emotion and negative emotion) to the tester as visual stimuli, the tester is required to freely watch in two image regions for a certain time, and eye movement track data of the tester is recorded in the watching process to construct a corresponding data set.
It should be noted that the above-mentioned visual saliency experimental paradigm fully combines the idea of image semantics, and in the competition of a pair of positive and negative emotion images, a tester may browse a region of interest according to his own attention mechanism, and the image region has certain image semantics, such as various semantic features like saliency distribution, emotion distribution, and the like. By analyzing the complete eye movement track information of the testee in the experimental process and combining the image semantic analysis means, the attention mechanism of the testee can be known, and the difference between normal people and depression patients can be explored.
In step S102, a significant fixation difference map of the normal population and the depressed population is obtained according to the significant difference detection network and the eye movement trajectory data.
It can be understood that, in the embodiment of the present application, the eye movement trajectory data of the tester for 160 emotion images in step S101 may be obtained, and a normal population fixation profile, a depressed population fixation profile, and two population fixation difference profiles corresponding to the 160 emotion images are generated, respectively, where each fixation profile of a part of emotion images is as shown in fig. 4.
Optionally, in some embodiments, deriving a significant gaze disparity map for the normal population and the depressed population based on the significant disparity detection network and the eye movement trajectory data comprises: obtaining a watching distribution diagram of the normal person for the emotional image through the significance detection network of the upper side branch and the eye movement track data; obtaining a watching distribution map of depressed people to the emotional image through the saliency detection network and the eye movement track data of the lower side branches; and splicing 64-channel significance characteristic graphs generated by the convolution operation of the upper branch and the lower branch by 3 multiplied by 3 to obtain a 128-channel significance characteristic graph, and obtaining a significance fixation difference distribution map of a normal population and a depressed population according to the 128-channel significance characteristic graph and the eye movement track data.
Specifically, in order to better predict the gaze difference distribution of two populations relative to the emotion image, the embodiment of the present application employs a three-branch significance difference detection network, as shown in fig. 5, the significance detection network of the upper branch is used to predict the gaze distribution of normal people to the emotion image, the significance detection network of the lower branch is used to predict the gaze distribution of depression patients to the emotion image, and 64-channel significance feature maps generated by performing convolution operation on the two branches by 3 × 3 are spliced to obtain a 128-channel significance feature map, which is used to predict a middle branch of the gaze difference distribution, and the upper branch and the lower branch respectively learn the gaze laws of the two populations to assist the middle branch in learning the gaze difference of the two populations, so that the whole network is expected to be more suitable for mining semantic regions with low interpersonal attention.
Therefore, by learning the gazing difference of the two crowds, the network can dig out the attention mechanism difference of the two crowds. The final purpose of the application is to identify the depression state of a tester, so that the fixation point of the tester on an emotion image can be calculated according to the eye movement track data of the tester on the emotion image in the experimental process, and the significance difference characteristics of the corresponding positions are extracted to represent the psychological state of the tester, thereby completing the task of identifying the depression state.
In step S103, a distribution feature map of the emotion image in each emotion dimension is generated through a preset weak supervision coupling network and the emotion image.
Optionally, in some embodiments, generating a distribution feature map of the emotion image in each emotion dimension through a preset weakly supervised coupling network and the emotion image includes: acquiring deep network characteristics corresponding to the positive emotion image and the negative emotion image; determining an emotion distribution prediction branch according to the deep network characteristics; and generating a distribution characteristic diagram of the emotion image in each emotion dimension according to the emotion distribution prediction branch and the over-preset weak supervision coupling network.
Optionally, in some embodiments, the sentiment dimensions include: eight emotional dimensions of entertainment, anger, feast, satisfaction, disgust, excitement, fear, and sadness.
It can be understood that the embodiment of the present application can adopt the weakly supervised coupling network structure as shown in fig. 6 to learn the emotion classification of the prediction image and generate the distribution of the image in each emotion dimension.
Specifically, in the embodiment of the present application, the network FCN ResNet-101 may be selected as a basic network module, and is used to extract deep network features corresponding to an image, and finally obtain a feature map with dimension reduced by 32 times compared with the original size (the original size is 448 × 448, the feature map size is 14 × 14, and the number of channels is 2048). The next network is divided into two branches for emotion distribution prediction and emotion class prediction, respectively.
And respectively generating a distribution characteristic diagram (comprising eight emotion dimensions of entertainment, anger, fearfulness, satisfaction, disgust, excitement, fear and sadness) of the emotion images in the visual significance experimental paradigm in each emotion dimension by using the trained weak supervision coupling network.
Therefore, the distribution characteristic diagram of the image corresponding to each emotion dimension can be obtained. The final purpose of the method is to identify the depression state of the tester, so that the corresponding fixation point can be calculated according to the eye movement track data of the image of the tester in the experimental process, and the distribution characteristics of the emotion dimensions at the corresponding positions are extracted to represent the psychological state of the tester, thereby completing the task of identifying the depression state.
In step S104, the difference between the attention mechanism of the normal population and the attention mechanism of the depressed population is obtained according to the distribution feature map of the significant gazing difference and the emotion dimension, so as to perform the depression state identification according to the difference between the attention mechanism.
It can be understood that, as shown in fig. 7, the embodiment of the application analyzes the difference of attention mechanism of two populations by collecting the eye movement trajectory data of the testers in the free viewing mode, combining the image semantic thought, and extracting the characteristics of the significant difference, emotion distribution and the like of the testers in the experimental process, and realizes the task of identifying the depression state. Through verification, the embodiment of the application obtains the depressive state identification precision of 92.42% on the corresponding data set. According to the method for identifying the eye movement of the depressed people based on the image significance difference and emotion analysis, the eye movement track data sets of normal people and the depressed people are constructed by designing a visual significance experimental paradigm combined with the image semantic thought, the difference of the two people in the attention mechanism is analyzed by extracting the significance difference characteristic and the emotion distribution characteristic of the experimental image, and the classification task of the two people is further realized. The free-viewing experimental paradigm adopts a combination of a positive emotion image and a negative emotion image as visual stimuli, a tester is required to freely view in an image area, and the whole process simulates a behavior pattern of the tester when the tester freely views in daily life.
Next, a device for identifying eye movements of depressed people based on analysis of significant differences and moods of images according to an embodiment of the present application will be described with reference to the accompanying drawings.
Fig. 8 is a block schematic diagram of a device for identifying eye movements of depressed people based on significant difference and emotion analysis of images according to an embodiment of the present application.
As shown in fig. 8, the device 10 for identifying the eye movement of the depressed people based on the significant difference and emotion analysis of the images includes: an acquisition module 100, an acquisition module 200, a generation module 300, and an identification module 400.
The acquisition module 100 is configured to acquire eye movement trajectory data of a tester within a first preset time. The obtaining module 200 is configured to obtain a significant gazing disparity map of a normal population and a depressed population according to the significant disparity detection network and the eye movement trajectory data. The generating module 300 is configured to generate a distribution feature map of the emotion image in each emotion dimension through a preset weak supervision coupling network and the emotion image. The identification module 400 is configured to obtain the difference between the attention mechanism of the normal group and the attention mechanism of the depressed group according to the distribution feature map of the significant gazing difference and the emotion dimension, so as to perform the depression state identification according to the difference between the attention mechanism.
Optionally, in some embodiments, the acquisition module 100 is specifically configured to: presenting a black background image on the screen, and enabling a tester to observe the screen within a second preset time so as to relieve the current attention of the tester; a white cross is displayed in the right center of the screen, and the tester concentrates the sight on the right center of the screen for a third preset time; presenting a first group of positive emotion images and negative emotion images on a screen, and continuing for a fourth preset time; observing any area of the first group of positive emotion images and the negative emotion images by a tester; and returning to execute the step of presenting a black background image on the screen, and observing the screen by the tester within a second preset time to relieve the current attention of the tester until the positive emotion images and the negative emotion images presented on the screen reach a preset group number.
Optionally, in some embodiments, the obtaining module 200 is specifically configured to: obtaining a watching distribution diagram of the normal person for the emotional image through the significance detection network of the upper side branch and the eye movement track data; obtaining a watching distribution map of depressed people to the emotional image through the saliency detection network and the eye movement track data of the lower side branches; and splicing 64-channel significance characteristic graphs generated by the convolution operation of the upper branch and the lower branch by 3 multiplied by 3 to obtain a 128-channel significance characteristic graph, and obtaining a significance fixation difference distribution map of a normal population and a depressed population according to the 128-channel significance characteristic graph and the eye movement track data.
Optionally, in some embodiments, the generating module 300 is specifically configured to: acquiring deep network characteristics corresponding to the positive emotion image and the negative emotion image; determining an emotion distribution prediction branch according to the deep network characteristics; and generating a distribution characteristic diagram of the emotion image in each emotion dimension according to the emotion distribution prediction branch and the over-preset weak supervision coupling network.
Optionally, in some embodiments, the sentiment dimensions include: eight emotional dimensions of entertainment, anger, feast, satisfaction, disgust, excitement, fear, and sadness.
It should be noted that the foregoing explanation of the embodiment of the method for identifying the eye movement of a depressed person based on the analysis of the significant difference and the emotion of the image is also applicable to the device for identifying the eye movement of a depressed person based on the analysis of the significant difference and the emotion of the image of the embodiment, and the details are not repeated here.
According to the depressed crowd eye movement identification device based on the image significance difference and emotion analysis, the eye movement track data sets of normal people and depressed patients are constructed by designing a visual significance experiment paradigm combined with image semantic ideas, the difference of the two crowds in an attention mechanism is analyzed by extracting the significance difference characteristic and emotion distribution characteristic of an experiment image, and classification tasks of the two crowds are further realized. The free-viewing experimental paradigm adopts a combination of a positive emotion image and a negative emotion image as visual stimuli, a tester is required to freely view in an image area, and the whole process simulates a behavior pattern of the tester when the tester freely views in daily life.
In the description herein, reference to the description of the term "one embodiment," "some embodiments," "an example," "a specific example," or "some examples," etc., means that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the application. In this specification, the schematic representations of the terms used above are not necessarily intended to refer to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or N embodiments or examples. Furthermore, various embodiments or examples and features of different embodiments or examples described in this specification can be combined and combined by one skilled in the art without contradiction.
Furthermore, the terms "first", "second" and "first" are used for descriptive purposes only and are not to be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one such feature. In the description of the present application, "N" means at least two, e.g., two, three, etc., unless specifically limited otherwise.
Any process or method descriptions in flow charts or otherwise described herein may be understood as representing modules, segments, or portions of code which include one or more N executable instructions for implementing steps of a custom logic function or process, and alternate implementations are included within the scope of the preferred embodiment of the present application in which functions may be executed out of order from that shown or discussed, including substantially concurrently or in reverse order, depending on the functionality involved, as would be understood by those reasonably skilled in the art of implementing the embodiments of the present application.
The logic and/or steps represented in the flowcharts or otherwise described herein, e.g., an ordered listing of executable instructions that can be considered to implement logical functions, can be embodied in any computer-readable medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, processor-containing system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions. For the purposes of this description, a "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of the computer-readable medium would include the following: an electrical connection (electronic device) having one or N wires, a portable computer diskette (magnetic device), a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). Additionally, the computer-readable medium could even be paper or another suitable medium upon which the program is printed, as the program can be electronically captured, via for instance optical scanning of the paper or other medium, then compiled, interpreted or otherwise processed in a suitable manner if necessary, and then stored in a computer memory.
It should be understood that portions of the present application may be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, the N steps or methods may be implemented in software or firmware stored in a memory and executed by a suitable instruction execution system. If implemented in hardware, as in another embodiment, any one or combination of the following techniques, which are known in the art, may be used: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application specific integrated circuit having an appropriate combinational logic gate circuit, a Programmable Gate Array (PGA), a Field Programmable Gate Array (FPGA), or the like.
It will be understood by those skilled in the art that all or part of the steps carried by the method for implementing the above embodiments may be implemented by hardware related to instructions of a program, which may be stored in a computer readable storage medium, and when the program is executed, the program includes one or a combination of the steps of the method embodiments.
In addition, functional units in the embodiments of the present application may be integrated into one processing module, or each unit may exist alone physically, or two or more units are integrated into one module. The integrated module can be realized in a hardware mode, and can also be realized in a software functional module mode. The integrated module, if implemented in the form of a software functional module and sold or used as a stand-alone product, may also be stored in a computer readable storage medium.
The storage medium mentioned above may be a read-only memory, a magnetic or optical disk, etc. Although embodiments of the present application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present application, and that variations, modifications, substitutions and alterations may be made to the above embodiments by those of ordinary skill in the art within the scope of the present application.
Claims (10)
1. A method for identifying the eye movement of depressed people based on the significant difference and emotion analysis of images is characterized by comprising the following steps:
acquiring eye movement track data of a tester in a first preset time;
according to the significance difference detection network and the eye movement track data, obtaining significance fixation difference graphs of normal people and depressed people;
generating a distribution characteristic diagram of the emotion images in each emotion dimension through a preset weak supervision coupling network and the emotion images; and
and obtaining the difference of the attention mechanism of the normal population and the depressed population according to the significant gazing difference and the distribution characteristic diagram of the emotional dimension, so as to identify the depressed state according to the difference of the attention mechanism.
2. The method of claim 1, wherein the collecting eye movement trajectory data of the tester in a first preset time comprises:
presenting a black background image on a screen, and observing the screen by the tester within a second preset time to release the current attention of the tester;
a white cross appears in the right center of the screen, and the tester concentrates the sight on the right center of the screen for a third preset time;
the screen presents a first group of positive emotion images and negative emotion images and lasts for a fourth preset time;
the tester observes the first group of positive emotion images and any area of the negative emotion images;
and returning to execute the step of presenting a black background image on the screen, and observing the screen by the tester in a second preset time to relieve the current attention of the tester until the positive emotional images and the negative emotional images presented on the screen reach a preset group number.
3. The method of claim 1, wherein obtaining a significant gaze disparity map for normal versus depressed populations based on a significant disparity detection network and the eye trajectory data comprises:
obtaining a watching distribution diagram of the normal person for the emotional image through the significance detection network of the upper side branch and the eye movement track data;
obtaining a watching distribution diagram of the depressed people for the emotional image through a significance detection network of a lower branch and the eye movement track data;
and splicing 64-channel significance characteristic graphs generated by the convolution operation of the upper branch and the lower branch by 3 x 3 to obtain a 128-channel significance characteristic graph, and obtaining a significance fixation difference distribution map of the normal population and the depressed population according to the 128-channel significance characteristic graph and the eye movement track data.
4. The method of claim 1, wherein the generating of the distribution feature map of the emotion images in each emotion dimension through the preset weakly supervised coupling network and the emotion images comprises:
acquiring deep network characteristics corresponding to the positive emotion image and the negative emotion image;
determining an emotion distribution prediction branch according to the deep network characteristics;
and generating a distribution characteristic diagram of the emotion image in each emotion dimension according to the emotion distribution prediction branch and the over-preset weak supervision coupling network.
5. The method of claim 4, wherein the sentiment dimensions comprise:
eight emotional dimensions of entertainment, anger, feast, satisfaction, disgust, excitement, fear, and sadness.
6. An apparatus for identifying eye movements of depressed people based on significant difference and emotion analysis of images, comprising:
the acquisition module is used for acquiring eye movement track data of a tester in first preset time;
the acquisition module is used for detecting a network and the eye movement track data according to the significance difference to obtain a significance gazing difference diagram of normal people and depressed people;
the generating module is used for generating a distribution characteristic diagram of the emotion images in each emotion dimension through a preset weak supervision coupling network and the emotion images; and
and the identification module is used for obtaining the difference of attention mechanism between the normal population and the depressed population according to the significant gazing difference and the distribution characteristic diagram of the emotion dimension so as to identify the depression state according to the difference of attention mechanism.
7. The device according to claim 6, wherein the acquisition module is specifically configured to:
presenting a black background image on a screen, and observing the screen by the tester within a second preset time to release the current attention of the tester;
a white cross appears in the right center of the screen, and the tester concentrates the sight on the right center of the screen for a third preset time;
the screen presents a first group of positive emotion images and negative emotion images and lasts for a fourth preset time;
the tester observes the first group of positive emotion images and any area of the negative emotion images;
and returning to execute the step of presenting a black background image on the screen, and observing the screen by the tester in a second preset time to relieve the current attention of the tester until the positive emotional images and the negative emotional images presented on the screen reach a preset group number.
8. The apparatus of claim 6, wherein the obtaining module is specifically configured to:
obtaining a watching distribution diagram of the normal person for the emotional image through the significance detection network of the upper side branch and the eye movement track data;
obtaining a watching distribution diagram of the depressed people for the emotional image through a significance detection network of a lower branch and the eye movement track data;
and splicing 64-channel significance characteristic graphs generated by the convolution operation of the upper branch and the lower branch by 3 x 3 to obtain a 128-channel significance characteristic graph, and obtaining a significance fixation difference distribution map of the normal population and the depressed population according to the 128-channel significance characteristic graph and the eye movement track data.
9. The apparatus of claim 6, wherein the generating module is specifically configured to:
acquiring deep network characteristics corresponding to the positive emotion image and the negative emotion image;
determining an emotion distribution prediction branch according to the deep network characteristics;
and generating a distribution characteristic diagram of the emotion image in each emotion dimension according to the emotion distribution prediction branch and the over-preset weak supervision coupling network.
10. The apparatus of claim 9, wherein the emotion dimensions comprise:
eight emotional dimensions of entertainment, anger, feast, satisfaction, disgust, excitement, fear, and sadness.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011530259.9A CN112674770B (en) | 2020-12-22 | 2020-12-22 | Depression crowd eye movement identification method based on image significance difference and emotion analysis |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011530259.9A CN112674770B (en) | 2020-12-22 | 2020-12-22 | Depression crowd eye movement identification method based on image significance difference and emotion analysis |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112674770A true CN112674770A (en) | 2021-04-20 |
| CN112674770B CN112674770B (en) | 2021-09-21 |
Family
ID=75450706
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011530259.9A Active CN112674770B (en) | 2020-12-22 | 2020-12-22 | Depression crowd eye movement identification method based on image significance difference and emotion analysis |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112674770B (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113658697A (en) * | 2021-07-29 | 2021-11-16 | 北京科技大学 | Psychological assessment system based on video fixation difference |
| CN115607159A (en) * | 2022-12-14 | 2023-01-17 | 北京科技大学 | Depression state identification method and device based on eye movement sequence space-time characteristic analysis |
Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101980248A (en) * | 2010-11-09 | 2011-02-23 | 西安电子科技大学 | Improved visual attention model-based method of natural scene object detection |
| US20140148728A1 (en) * | 2012-11-20 | 2014-05-29 | El-Mar Inc. | Method of identifying an individual with a disorder or efficacy of a treatment of a disorder |
| CN105559802A (en) * | 2015-07-29 | 2016-05-11 | 北京工业大学 | Tristimania diagnosis system and method based on attention and emotion information fusion |
| CN108427929A (en) * | 2018-03-19 | 2018-08-21 | 兰州大学 | A kind of depressed discriminance analysis system based on tranquillization state brain network |
| CN108932532A (en) * | 2018-07-11 | 2018-12-04 | 杭州电子科技大学 | A kind of eye movement data number suggesting method required for the prediction of saliency figure |
| CN109199412A (en) * | 2018-09-28 | 2019-01-15 | 南京工程学院 | Abnormal emotion recognition methods based on eye movement data analysis |
| WO2019227690A1 (en) * | 2018-06-01 | 2019-12-05 | 中国科学院上海生命科学研究院 | Screening of behavioral paradigm indicators and application thereof |
| CN110764613A (en) * | 2019-10-15 | 2020-02-07 | 北京航空航天大学青岛研究院 | Eye movement tracking calibration method based on head-mounted eye movement module |
| CN111967363A (en) * | 2020-08-10 | 2020-11-20 | 河海大学 | Emotion prediction method based on micro-expression recognition and eye movement tracking |
| EP3748649A1 (en) * | 2018-12-14 | 2020-12-09 | Shenzhen Institutes of Advanced Technology | Method for evaluating multi-modal emotional understanding capability of patient with autism spectrum disorder |
-
2020
- 2020-12-22 CN CN202011530259.9A patent/CN112674770B/en active Active
Patent Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101980248A (en) * | 2010-11-09 | 2011-02-23 | 西安电子科技大学 | Improved visual attention model-based method of natural scene object detection |
| US20140148728A1 (en) * | 2012-11-20 | 2014-05-29 | El-Mar Inc. | Method of identifying an individual with a disorder or efficacy of a treatment of a disorder |
| CN105559802A (en) * | 2015-07-29 | 2016-05-11 | 北京工业大学 | Tristimania diagnosis system and method based on attention and emotion information fusion |
| CN108427929A (en) * | 2018-03-19 | 2018-08-21 | 兰州大学 | A kind of depressed discriminance analysis system based on tranquillization state brain network |
| WO2019227690A1 (en) * | 2018-06-01 | 2019-12-05 | 中国科学院上海生命科学研究院 | Screening of behavioral paradigm indicators and application thereof |
| CN108932532A (en) * | 2018-07-11 | 2018-12-04 | 杭州电子科技大学 | A kind of eye movement data number suggesting method required for the prediction of saliency figure |
| CN109199412A (en) * | 2018-09-28 | 2019-01-15 | 南京工程学院 | Abnormal emotion recognition methods based on eye movement data analysis |
| EP3748649A1 (en) * | 2018-12-14 | 2020-12-09 | Shenzhen Institutes of Advanced Technology | Method for evaluating multi-modal emotional understanding capability of patient with autism spectrum disorder |
| CN110764613A (en) * | 2019-10-15 | 2020-02-07 | 北京航空航天大学青岛研究院 | Eye movement tracking calibration method based on head-mounted eye movement module |
| CN111967363A (en) * | 2020-08-10 | 2020-11-20 | 河海大学 | Emotion prediction method based on micro-expression recognition and eye movement tracking |
Non-Patent Citations (3)
| Title |
|---|
| PHILIPPE FOSSATI: "Circuit based anti-correlation, attention orienting and major depression", 《CNS SPECTRUMS》 * |
| ZEYU PAN: "Depression Detection Based on Reaction Time and Eye Movement", 《2019 IEEE INTERNATIONAL CONFERENCE ON IMAGE PROCESSING》 * |
| 纵宝宝: "立体图像显著性检测模型研究", 《中国优秀硕士学位论文全文数据库》 * |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113658697A (en) * | 2021-07-29 | 2021-11-16 | 北京科技大学 | Psychological assessment system based on video fixation difference |
| CN113658697B (en) * | 2021-07-29 | 2023-01-31 | 北京科技大学 | Psychological assessment system based on video fixation difference |
| CN115607159A (en) * | 2022-12-14 | 2023-01-17 | 北京科技大学 | Depression state identification method and device based on eye movement sequence space-time characteristic analysis |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112674770B (en) | 2021-09-21 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106691476B (en) | Image cognition psychoanalysis system based on eye movement characteristics | |
| Moody et al. | More than mere mimicry? The influence of emotion on rapid facial reactions to faces. | |
| US7942816B2 (en) | Psychotic manifestation and mental state evaluation apparatus and evaluation method | |
| JP6426105B2 (en) | System and method for detecting blink suppression as a marker of engagement and sensory stimulus saliency | |
| Chaouachi et al. | Affect and mental engagement: Towards adaptability for intelligent | |
| EP2094159B1 (en) | System and method for detecting a specific cognitive-emotional state in a subject | |
| CN109770921B (en) | Method and device for screening early stage language and cognitive ability of autistic children | |
| CN111326253A (en) | Method for evaluating multi-modal emotional cognitive ability of patients with autism spectrum disorder | |
| CN112674770B (en) | Depression crowd eye movement identification method based on image significance difference and emotion analysis | |
| CN108065942B (en) | Method for compiling stimulation information aiming at oriental personality characteristics | |
| Sakaki et al. | Association learning for emotional harbinger cues: When do previous emotional associations impair and when do they facilitate subsequent learning of new associations? | |
| Laws | Penile plethysmography: Strengths, limitations, innovations | |
| KR101854812B1 (en) | Psychiatric symptoms rating scale system using multiple contents and bio-signal analysis | |
| CN112133407A (en) | Rapid intelligent emotion assessment analysis method based on voice and expression | |
| McDowell et al. | Blending qualitative and quantitative study methods in health services research | |
| JP3954295B2 (en) | IDENTIFICATION / RESPONSE MEASUREMENT METHOD, COMPUTER-READABLE RECORDING MEDIUM CONTAINING IDENTIFICATION / REACTION MEASUREMENT PROGRAM | |
| CN114209324A (en) | Psychological assessment data acquisition method based on image visual cognition and VR system | |
| CN111317448B (en) | Method and system for analyzing visual space cognition | |
| CN112674771A (en) | Depression crowd identification method and device based on image fixation difference | |
| CN113658697B (en) | Psychological assessment system based on video fixation difference | |
| CN112890815A (en) | Autism auxiliary evaluation system and method based on deep learning | |
| CN103251418B (en) | Image cognition psychoanalysis system | |
| CN211862821U (en) | Autism auxiliary evaluation system based on deep learning | |
| CN110840467A (en) | Correlation analysis method for eye movement data and mental system diseases | |
| Lerner | Knowledge or performance—Why youth with autism experience social problems |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |