CN106681484B - In conjunction with the image object segmenting system of eye-tracking - Google Patents
In conjunction with the image object segmenting system of eye-tracking Download PDFInfo
- Publication number
- CN106681484B CN106681484B CN201510751861.8A CN201510751861A CN106681484B CN 106681484 B CN106681484 B CN 106681484B CN 201510751861 A CN201510751861 A CN 201510751861A CN 106681484 B CN106681484 B CN 106681484B
- Authority
- CN
- China
- Prior art keywords
- image
- data
- user
- eye movement
- eye
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Expired - Fee Related
Links
Landscapes
- Measurement And Recording Of Electrical Phenomena And Electrical Characteristics Of The Living Body (AREA)
- Image Processing (AREA)
Abstract
The present invention devises a kind of image object segmenting system of combination eye-tracking, can split the interested image object of user in the image that a width user watches.When user watches computer screen, using the eye movement data of eye-tracking instrument acquisition user, while the EEG signals of eeg recording instrument acquisition user are used；Eye movement data is used to analyze the eye movement of user；EEG signals are for analyzing whether user's brain is in states of interest in real time；The Conjoint Analysis of eye movement and user's brain states can reveal that user to which area interest in image, so that it is determined that in image target object position；Then by the super-pixel segmentation of image procossing with merge algorithm, the interested target object of user is analyzed and is divided, target object is divided from image and is extracted, for the purposes such as searching for, matching.
Description
Technical field
The present invention provides a kind of eye movement and eeg data watched in image process based on user, determines the target in image
Object and the method for being split target object from image using computer image processing technology.It refers specifically to watch as user
When computer screen, using the eye movement data of eye-tracking instrument acquisition user, while the brain of eeg recording instrument acquisition user is used
Electric signal；Eye movement data is used to analyze the eye movement of user；EEG signals are for analyzing whether user's brain is in sense in real time
Interest state；The Conjoint Analysis of eye movement and user's brain states can reveal that user to which area interest in image,
So that it is determined that in image target object position；Then by the super-pixel segmentation of image procossing with merge algorithm, to user's sense
The target object of interest is analyzed and is divided, and target object is divided from image and is extracted, and uses for searching for, matching etc.
On the way.The invention belongs to the connected applications of Cognitive Neuroscience, signal processing technology and image procossing, lead for automatic control technology
Domain.
Background technique
With the arriving of Internet era, by text, image, sound and view in such a way that text transmits information
Replaced the multimedia modes such as frequency.For the mankind, the information of vision transmitting has accounted for the overwhelming majority, and image has become this
Important information carrier of a epoch.Segmentation for target object in image emerges diversified algorithm in recent years.
In image partition method, water is used in conjunction with the methods of sparse neighbour propagation and the Study Of Segmentation Of Textured Images of quick spectral clustering
The dividing method of flat collection, is split with energy spectrometer, be ensure that splitting speed but be can be only applied to fixed mode image point
It cuts, the similar professional domain such as the structure of medical image is not suitable for the segmentation of heterogeneous image on internet.Based on super-pixel and
The level set image segmentation method that figure cuts optimization carries out image segmentation using the derivative algorithm that neighbour clusters, and can preferably keep
The region consistency of texture image, and computation complexity is low, but it is not suitable for that boundary value is fuzzy, image segmentation of similar gray value,
This method is very sensitive to threshold value simultaneously.
There are also some image segmentation algorithms to improve the Target Segmentation performance in image using the interactive information of user, such as
GrubCut algorithm and TouchCut algorithm.But these algorithms do in image target object when divide, exist it is many not
Foot such as needs to rely on complicated interactive information, needs to add a large amount of artificial markup information；Mesh in image cannot be handled well
Target scale problem, the problem of being difficult resolution image prospect and background.
As it can be seen that improving interactive mode becomes an important method for promoting image segmentation system performance.The present invention utilizes eye
Dynamic rail mark completes interactive process with two kinds of eeg data emerging man-machine interaction modes.With using mouse-keyboard or and press touch
Traditional interactive mode of screen is compared, and the present invention directly passes through the eye movement and the electric two kinds of physiological signals of brain of people, opens Article 2 people
The region of interest of user is passed to computer real-time, quickly by machine interaction channel.This interactive process is natural, for user just
It is small in the work interference of progress, while interactive information is genuine and believable.The present invention has wide application for the industries such as designing, investigating
Prospect.
Summary of the invention
Fig. 1 is the composition schematic diagram of the image object segmenting system in conjunction with eye movement and eeg data.
Fig. 2 is the algorithm flow chart of the image object segmenting system in conjunction with eye movement and eeg data.
The purpose of the present invention is realizing a kind of new image segmentation system, the image on computer screen is watched using user
When synchronous acquisition eye movement and eeg data, extract the interested information on target object of user, improve image object segmentation property
Energy.Eye-tracking instrument and eeg recording instrument synchronous acquisition eye movement and eeg data are utilized in the present invention, can be analyzed in eye movement data
The picture position point that each moment user is seeing out, we term it focus；Eeg data may determine that user is same simultaneously
One moment user's brain is to the interest level to the image-region seen, we term it attention rates.By eye movement data and brain
After electric data carry out Conjoint Analysis, so that it may obtain the focus and attention rate of user, determine user to which region in image
Content it is interested.The region of interesting target object or location information in image are provided by this mode user, we
Referred to as target area.
After obtaining the target area information in image, super-pixel segmentation is first carried out to the image of viewing, then utilizes mesh
Mark area information merges the super-pixel of segmentation, retains interesting part after merging, and will lose interest in part rejecting, i.e.,
The segmentation of target object in image can be achieved.The image object segmentation of this mode compares traditional image segmentation mode and increases
As supervision message, accuracy is higher for eye movement and eeg data, while avoiding user and needing specified cut zone, shape manually
And etc., it can effectively improve segmentation precision, while the other work currently carried out with user do not interfere with mutually, actually answering
It is more objective reliable with the segmentation result of middle target object.
The present invention includes following module:
(1) eye movement and brain electric data collecting module: acquiring eye movement by eye-tracking instrument, dry by eeg recording instrument
Electrode acquires eeg data, while timestamp is added, and is convenient for eye movement data convenient for the Conjoint Analysis of eye movement data and eeg data
With eeg data synchronous acquisition and Conjoint Analysis.
(2) hot spot, clustering analysis data aggregate analysis module: are carried out to obtain the visual attention location point of user to eye movement data
Clustering carries out analysis to eeg data and obtains the attention force information of user simultaneously to obtain the visual attention location point of user
The attention rate that analysis obtains user is carried out to eeg data,；Then the two is combined, user can be obtained in attention collection
In in the case where the region paid close attention to can obtain user and feel the position of the target object of emerging ring interest in the picture, i.e. user is really closed
The target area information area of note, target area information will be used to instruct to get off the concern information preservation of user so as to super-pixel
Merging when use.
(3) image processing module: in acquisition user's eye movement, on backstage, the image to viewing carries out first with when brain electric information
Material carries out super-pixel segmentation, after having got the attention and information of user, is believed according to the target area attention of user
Breath carries out super-pixel merging, while rejecting the picture material not being concerned, and retains the picture material being concerned.
Potential application of the invention has:
(1) detection and extraction for realizing sensitive target in image, can be used for the monitoring of video or image, realize it is sensitive or
The detection and extraction of important goal special body size.
(2) information retrieval and push are realized, can be used for collecting user and watch interested content during image or video,
And it retrieves similar image or video from network accordingly and is pushed to user.
(3) a kind of mode of evaluation is provided, is more attracted for evaluating which part in the designed images such as building and advertisement
The attention of people.
(4) it is used for clinical assistant diagnosis, for certain aided diagnosis methods with eyeball or state of mind related disease, such as
Nystagmus and schizophrenia etc..
This system includes that the specific embodiment of three modules is as follows:
(1) eye movement and brain electric data collecting module
When image is presented on computer screen, eye movement and brain electric data collecting module synchronization record eye movement and brain electricity number
According to.Eye movement data acquisition uses the X120 eye-tracking instrument of Tobii company, and eeg data uses NeuroSky company
MindWave Mobile portable brain electric recorder.Eye-tracking instrument and eeg recording view in real time transmit the data of acquisition
To same computer.A data acquisition is realized on eye-tracking instrument and the driving of eeg recording instrument on computer
Synchronization module, the eye movement data that computer real-time reception is arrived and eeg data add timestamp information, i.e., in computer
On to eye movement data and eeg data add synchronizing information.
(2) data aggregate analysis module
Data aggregate analysis module is the data processor run on computers, mainly includes three parts: data
Pretreatment, focus and attention-degree analysis, the marginal analysis of target area.
Data prediction is filtered to eye movement data and eeg data, and the filtering etc. including eye movement and eeg data is gone
Noisy operation, and from reconstructed in the eye movement data of images of left and right eyes user watch computer screen on image process in eye
Dynamic rail mark etc..Wherein noise be eye-tracking instrument acquisition eye movement data in include twitching of the eyelid point and pan point, these eyes
Dynamic data can interfere with blinkpunkt cluster as a result, so being filtered out.The data of eye-tracking instrument acquisition are images of left and right eyes point
From data, there are parallaxes for right and left eyes when human eye is look at same object, this is the mankind it can be seen that three-dimension object
Basis, but when parallax of eyes is look at flat image, can have an impact blinkpunkt cluster result, so locating in advance in data
When reason will use binocular parallax registration Algorithm the parallax of eyes be eliminated, obtain the coordinate of true binocular fixation point, in conjunction with when
Between stamp signal reconstruct go out user watch computer on image when eye movement.
Focus and attention-degree analysis include two aspects: it is on the one hand the analysis of rhythm and pace of moving things wave energy is carried out to eeg data,
The attention rate of user is calculated, i.e., whether each moment brain is in states of interest；It on the other hand is to the attention rate higher moment
Corresponding eye movement carries out clustering, determines the position of the interested image object of user.
In attention-degree analysis, the energy of rhythm and pace of moving things wave in order to obtain, it is necessary first to which power Spectral Estimation is carried out to EEG signals.This
The method of invention power Spectral Estimation is as follows: (1) using data in sliding window technique intercepts window, the size of neutron window is 125
Point (brain electricity sample frequency be 256 hertz) has 2/3 overlapping between sub- window；(2) by sub- window data padding, 256 points are extended to,
Period map method is recycled to carry out power Spectral Estimation to the data in each window；(3) average value of each sub- window power spectrum is found out, this
A average value is required power spectral density.The present invention finds out the energy of rhythm and pace of moving things wave according to the power spectral density at each moment,
The rhythm and pace of moving things wave energy of 5 frequency ranges of 0-100Hz is expressed as E (δ), E (θ), E (α), E (β), E (γ).Wherein rhythm and pace of moving things wave frequency
Section δ wave is 0-4Hz, and θ wave is 4-8Hz, and α wave is 8-12Hz, β 12-30Hz, γ 30-100Hz.Attention rate can by ratio (
E (α)+E (β))/E (θ) measured.
It pays close attention in point analysis, the characteristics of extremely dispersion for the relatively high eye movement data of attention rate, is calculated using cluster
Method separates the eye movement point for watching different target attentively, forms several comparison eye movement accumulation point areas around image object
Domain.The present invention uses the Hybrid Clustering Algorithm of a Density Clustering and K-means.When can be relatively high by attention rate by cluster
It carves corresponding eye movement point and is divided into several accumulation regions, the eye movement point in each accumulation regions is corresponded to divided image
Target, the accumulation regions of eye movement point are the target area paid close attention to.
The marginal analysis of target area is the boundary that go out image object according to a preliminary estimate.Eye movement aggregation by cluster
Point in area can distinguish the target object of user's concern, but tracing point all concentrates on target object mostly after clustering
Inside is unfavorable for analyzing the profile of target object.The present invention is found using algorithm of convex hull and entirely clusters corresponding image object
Edge contour.
(3) image processing module
Image processing module is the program handled image run on computers, including two parts: super picture
Element segmentation, super-pixel merge to be divided with target image.
Super-pixel segmentation, which refers to, divides the image into small pixel block.Super-pixel segmentation part present invention uses SLIC algorithm,
Image is transformed into CIE-Lab color space from RGB color first by SLIC algorithm, by (L, a, b) face of each pixel
Color value and (X, Y) coordinate value as a five dimensional vector V [L, a, b, X, Y], the similitude of two pixels just use this to
Span is from measuring.All pixels are then repeatedly scanned with by the seed point generated, each pixel is divided into it recently
The same cluster of seed.It can be seen that SLIC is a kind of method based on color and apart from progress super-pixel segmentation, therefore its partitioning scheme
Very similar to the true viewing mode of people, a focus is found in the picture first, then expand from the focus and regard
Find entire interested image object in open country.
Super-pixel merging is to obtain the interactive information of user in eye movement and eeg data Conjoint Analysis module, interaction here
Information is exactly the concern target area characterized with the accumulation regions of image object, instructs super-pixel to merge using these interactive information,
The super-pixel block that eye movement point in accumulation regions is passed through merges, and forming quantity is less, the biggish block of pixels of area.
Target image segmentation refer to super-pixel close close obtain biggish block of pixels after, reservation include need divide it is complete
Image object part, extra part is removed, to complete the segmentation of image object.
Claims (1)
1. combining the image object segmenting system of eye-tracking, this set system includes:
(1) computer screen data acquisition module: is watched by eye-tracking instrument and portable brain electric recorder synchronous acquisition user
Eye movement and eeg data when image on curtain, collect the interactive information of user, improve the performance of image object segmentation；Described
Data acquisition module block feature include: eye movement data and eeg data real-time acquisition and two class real-time data transmissions to calculate
Upper addition synchronized timestamp realizes the synchronization of data；
The data of acquisition are transferred to same computer in real time by eye-tracking instrument and eeg recording instrument；Eye movement on computer
The synchronization module that the acquisition of a data is realized on the driving of tracker and eeg recording instrument, computer real-time reception is arrived
Eye movement data and eeg data add timestamp information, i.e., synchronous with eeg data addition to eye movement data on computers
Information；
(2) data aggregate analysis module: running eye movement on computers and eeg data analyzes program, by the brain electricity of acquisition with
After eye movement data carries out basic pretreatment, automatically from eye movement data analyze user watch image eye movement, eye movement with
The data of track instrument acquisition are the data of images of left and right eyes separation, right and left eyes when human eye is look at same object there are parallax, this
The mankind it can be seen that three-dimension object basis, but when parallax of eyes is look at flat image, can be to blinkpunkt cluster result
It has an impact, so to use binocular parallax registration Algorithm to eliminate the parallax of eyes in data prediction, obtains true
The coordinate of binocular fixation point, binding time stamp signal reconstruct go out eye movement of the user in the image watched on computer；Together
When the interest level of each moment user is analyzed from eeg data, and it is user's interest level higher moment is corresponding
Eye movement point data carry out clustering, determine image target area, finally find image using convex closure profile algorithm
The boundary of target；The feature of the data aggregate analysis module includes: every using brain wave rhythm wave energy analysis user's brain
The states of interest at a moment, the eye movement of each moment user is positioned using eye movement data, and the two binding analysis user sees
It sees the interested eye movement point distribution in image process, and further determines that user is interested using cluster and algorithm of convex hull
Image target area and boundary；
(3) image processing module: running image processing program on computers, image is first carried out super-pixel segmentation, then
The image target area information that eye movement is obtained with eeg data analysis module is used to super-pixel to merge, user is interested
Image object split；The feature of the image processing module includes: the image object area obtained using Conjoint Analysis
Domain information instructs super-pixel merging process and then realizes the segmentation of target image.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510751861.8A CN106681484B (en) | 2015-11-06 | 2015-11-06 | In conjunction with the image object segmenting system of eye-tracking |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510751861.8A CN106681484B (en) | 2015-11-06 | 2015-11-06 | In conjunction with the image object segmenting system of eye-tracking |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN106681484A CN106681484A (en) | 2017-05-17 |
| CN106681484B true CN106681484B (en) | 2019-06-25 |
Family
ID=58862870
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201510751861.8A Expired - Fee Related CN106681484B (en) | 2015-11-06 | 2015-11-06 | In conjunction with the image object segmenting system of eye-tracking |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106681484B (en) |
Families Citing this family (16)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107168134B (en) * | 2017-06-27 | 2020-08-11 | 福州大学 | Brain-eye hybrid driven household appliance control method |
| CN107888973B (en) | 2017-11-24 | 2019-12-10 | 华南理工大学 | Electroencephalogram controlled video input auditory display blind guiding device and method |
| CN108459705A (en) * | 2017-12-21 | 2018-08-28 | 北京数字新思科技有限公司 | A kind of method, apparatus and computer readable storage medium of test image attraction |
| CN108419011A (en) * | 2018-02-11 | 2018-08-17 | 广东欧珀移动通信有限公司 | Image pickup method and Related product |
| CN108492648A (en) * | 2018-03-16 | 2018-09-04 | 何戴娆 | A kind of remote online teaching student's state determines method and system |
| CN108874148A (en) * | 2018-07-16 | 2018-11-23 | 北京七鑫易维信息技术有限公司 | A kind of image processing method and device |
| CN109165646A (en) * | 2018-08-16 | 2019-01-08 | 北京七鑫易维信息技术有限公司 | The method and device of the area-of-interest of user in a kind of determining image |
| CN109271030B (en) * | 2018-09-25 | 2020-12-22 | 华南理工大学 | Multidimensional comparison method for three-dimensional space betting viewpoint track |
| CN109119161A (en) * | 2018-10-26 | 2019-01-01 | 首都医科大学附属北京安定医院 | A kind of schizoid diagnostic model |
| CN109901718A (en) * | 2019-03-07 | 2019-06-18 | 中国联合网络通信集团有限公司 | Data processing method and device, investigation system |
| CN110033652B (en) * | 2019-03-11 | 2021-06-04 | 中国民航大学 | Radar dynamic target calibration method and system based on eye tracking |
| CN112034977B (en) * | 2019-06-04 | 2022-07-19 | 陈涛 | Method for MR intelligent glasses content interaction, information input and recommendation technology application |
| CN110414377B (en) * | 2019-07-09 | 2020-11-13 | 武汉科技大学 | Remote sensing image scene classification method based on scale attention network |
| CN112613364A (en) * | 2020-12-10 | 2021-04-06 | 新华网股份有限公司 | Target object determination method, target object determination system, storage medium, and electronic device |
| WO2023004734A1 (en) * | 2021-07-30 | 2023-02-02 | Pricewaterhousecoopers Llp | Gaze predictor |
| CN114371781A (en) * | 2021-12-31 | 2022-04-19 | 金地（集团）股份有限公司 | User portrait generation method and system in real estate marketing |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101912263A (en) * | 2010-09-14 | 2010-12-15 | 北京师范大学 | Real-time functional magnetic resonance data processing system based on brain functional network component detection |
| CN103631941A (en) * | 2013-12-11 | 2014-03-12 | 北京师范大学 | Electroencephalogram-based target image retrieval system |
| CN104965584A (en) * | 2015-05-19 | 2015-10-07 | 西安交通大学 | Mixing method for brain-computer interface based on SSVEP and OSP |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20070060830A1 (en) * | 2005-09-12 | 2007-03-15 | Le Tan Thi T | Method and system for detecting and classifying facial muscle movements |
-
2015
- 2015-11-06 CN CN201510751861.8A patent/CN106681484B/en not_active Expired - Fee Related
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101912263A (en) * | 2010-09-14 | 2010-12-15 | 北京师范大学 | Real-time functional magnetic resonance data processing system based on brain functional network component detection |
| CN103631941A (en) * | 2013-12-11 | 2014-03-12 | 北京师范大学 | Electroencephalogram-based target image retrieval system |
| CN104965584A (en) * | 2015-05-19 | 2015-10-07 | 西安交通大学 | Mixing method for brain-computer interface based on SSVEP and OSP |
Non-Patent Citations (1)
| Title |
|---|
| "图像感兴趣区域提取方法研究";陈再良;《中国优秀博士学位论文全文数据库-信息科技辑》;20121215;论文第18-24、56-59页 |
Also Published As
| Publication number | Publication date |
|---|---|
| CN106681484A (en) | 2017-05-17 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106681484B (en) | In conjunction with the image object segmenting system of eye-tracking | |
| CN103340637B (en) | Move and driver's Alertness intelligent monitor system of brain electro' asion and method based on eye | |
| CN103164694B (en) | A kind of human action knows method for distinguishing | |
| Kim et al. | Saliency prediction on stereoscopic videos | |
| CN105147274A (en) | Method for extracting heart rate from visible spectrum section face video signal | |
| CN110353673B (en) | Electroencephalogram channel selection method based on standard mutual information | |
| CN110269600A (en) | Based on polynary empirical mode decomposition and the contactless video heart rate detection method for combining blind source separating | |
| CN106485208B (en) | The automatic removal method of eye electrical interference in single channel EEG signals | |
| CN107480716A (en) | Method and system for identifying saccade signal by combining EOG and video | |
| CN105701806A (en) | Depth image-based parkinson's tremor motion characteristic detection method and system | |
| CN108958489A (en) | A kind of interesting image regions Rapid Detection method based on brain electricity and eye tracker | |
| US20200294189A1 (en) | Information processing device, information processing method, recording medium storing program code, and information processing system | |
| CN110236515A (en) | A kind of contactless heart rate detection method based on near-infrared video | |
| CN104123549A (en) | Eye positioning method for real-time monitoring of fatigue driving | |
| Picot et al. | Comparison between EOG and high frame rate camera for drowsiness detection | |
| CN109255309A (en) | Brain electricity and eye movement fusion method and device towards Remote Sensing Target detection | |
| US10679394B2 (en) | Information processing device, information processing method, computer program product, and biosignal measurement system | |
| CN102662470B (en) | A kind of method and system realizing eye operation | |
| CN106713787A (en) | Evaluation method for watching comfort level caused by rolling subtitles of different speed based on EEG | |
| CN113743471A (en) | Driving evaluation method and system | |
| CN105700687B (en) | Single examination time brain electricity P300 component detection methods based on folding HDCA algorithms | |
| CN109276227A (en) | Based on EEG technology to visual fatigue analysis method caused by three-dimensional Depth Motion | |
| CN105468738A (en) | Image retrieval method based on combination of eye movement and electroencephalogram | |
| CN104063871A (en) | Method for segmenting image sequence scene of wearable device | |
| CN102932638A (en) | 3D video monitoring method based on computer modeling |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CF01 | Termination of patent right due to non-payment of annual fee | ||
| CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20190625 Termination date: 20201106 |