CN103500061A

CN103500061A - Method and equipment for controlling displayer

Info

Publication number: CN103500061A
Application number: CN201310445336.4A
Authority: CN
Inventors: 高伟
Original assignee: Samsung Electronics China R&D Center; Samsung Electronics Co Ltd
Current assignee: Samsung Electronics China R&D Center; Samsung Electronics Co Ltd
Priority date: 2013-09-26
Filing date: 2013-09-26
Publication date: 2014-01-08
Anticipated expiration: 2033-09-26
Also published as: CN103500061B

Abstract

Provided is a method and equipment for controlling a displayer. The method comprises the steps that a human eye image is obtained by shooting a human eye through electronic glasses; human eye feature information is extracted from the human eye image; the glasses position and the gesture information of the electronic glasses are detected; according to the human eye feature information, the glasses position and the gesture information, a human eye fixation point on a screen is detected; according to the human eye feature information and/or the changing situation of the human eye fixation point on the screen, human eye action is detected; displayer operation is controlled according to human eye action and the human eye fixation point on the screen. According to the method and equipment, the displayer can be controlled according to the human eye image.

Description

Control the method and apparatus of display

Technical field

The present invention relates to the control field of display, more particularly, relate to a kind of method and apparatus of controlling display.

Background technology

In people's life, display has become an indispensable part, no matter is television indicator, or computer monitor, and the user more and more wishes to control display by control method more easily.

Under existing technical conditions, the user controls display with the button on hand-held remote control device or display pannel usually, and this makes operating process be unable to do without the participation of hand, and people can't free hand from control device, especially for the people of both hands deformity, more inconvenient.

Therefore, need a kind of method and apparatus of controlling display, with can obstructed remarkable hand control display.

Summary of the invention

The object of the present invention is to provide a kind of method and apparatus of controlling display, it can control display according to eye image.

A first aspect of the present invention provides a kind of method of controlling display, comprising: take human eye by electronic glasses and obtain eye image; Extract human eye feature information from eye image; Glasses position and the attitude information of detected electrons glasses; Detect the human eye blinkpunkt on screen according to human eye feature information, glasses position and attitude information; Detect the human eye action according to the situation of change of the human eye blinkpunkt on human eye feature information and/or screen; Control the operation of display according to the human eye blinkpunkt on human eye action and screen.

Alternatively, according to the step of the human eye blinkpunkt on human eye feature information, glasses position and attitude information detection screen, comprise: according to the human eye blinkpunkt on human eye feature information acquisition electronic glasses; Obtain the human eye blinkpunkt on screen according to the human eye blinkpunkt on electronic glasses, glasses position and attitude information.

Alternatively, the positional information that described human eye feature information is pupil center and corneal reflection point comprises according to the step of the human eye blinkpunkt on human eye feature information acquisition electronic glasses: according to the positional information calculation of pupil center and corneal reflection point, go out the human eye blinkpunkt of people's an eye line on electronic glasses.

Alternatively, the action of described human eye comprise close one's eyes, watch attentively, at least one in twitching of the eyelid, smooth motion.

Alternatively, in the time of can't from human eye feature information, obtaining corneal reflection point within first schedule time, the human eye motion detection is for closing one's eyes; When the human eye blinkpunkt on screen remains unchanged within second schedule time, the human eye motion detection is for watching attentively; When the moving range within the 3rd schedule time of the human eye blinkpunkt on screen surpasses predetermined threshold, the human eye motion detection is twitching of the eyelid; When the human eye blinkpunkt on screen moves at a predetermined velocity, the human eye motion detection is smooth motion.

Alternatively, control the operation of display according to the human eye blinkpunkt on human eye action and screen under different patterns.

Alternatively, the step of controlling the operation of display according to the human eye blinkpunkt on human eye action and screen comprises: when display image screen, controlling display according to the human eye blinkpunkt on human eye action and screen proceeds as follows: when human eye moves as watching attentively, and, when the human eye blinkpunkt on screen is positioned at the screen zone line, near the image be positioned at the human eye blinkpunkt on screen is amplified and shown; In the situation that the image of screen display is moving image, when the human eye action is level and smooth the tracking, according to the human eye blinkpunkt on the screen in level and smooth tracing process, determine the object that human eye is followed the trail of, the object that human eye is followed the trail of is illustrated in the central authorities of screen; When human eye action, for watching attentively, and the human eye blinkpunkt on screen is positioned at screen border when regional, and image is moved towards the direction away from this border.

Alternatively, the step of controlling the operation of display according to the human eye blinkpunkt on human eye action and screen comprises: when the screen display user interface, controlling display according to the human eye blinkpunkt on human eye action and screen proceeds as follows: the human eye blinkpunkt on screen is positioned at a functional area, then human eye action when closing one's eyes, moves the function corresponding with this functional area; When human eye action is smooth motion, and the human eye blinkpunkt on screen is switched to this another zone by focus while from a functional area, moving to another functional area.

Alternatively, the method for described control display also comprises: the viewing angle while according to human eye feature information, glasses position and attitude information detection, watching screen; Adjust the degree of depth of screen according to viewing angle.

Alternatively, the step of adjusting the degree of depth of screen according to viewing angle comprises: when viewing angle surpasses the predetermined angular threshold value, the degree of depth in the zone of close electronic glasses on screen is turned down, the degree of depth away from the zone of electronic glasses on screen is tuned up, make viewing angle be less than or equal to the predetermined angular threshold value.

A second aspect of the present invention provides a kind of equipment of controlling display, comprising: the eye image acquiring unit, and take human eye by electronic glasses and obtain eye image; Feature extraction unit is extracted human eye feature information from eye image; The position and attitude detecting unit, glasses position and the attitude information of detected electrons glasses; The blinkpunkt detecting unit, detect the human eye blinkpunkt on screen according to human eye feature information, glasses position and attitude information; The human eye motion detection unit, detect the human eye action according to the situation of change of the human eye blinkpunkt on human eye feature information and/or screen; Control module, control the operation of display according to the human eye blinkpunkt on human eye action and screen.

Alternatively, the blinkpunkt detecting unit comprises: glasses blinkpunkt detecting unit, according to the human eye blinkpunkt on human eye feature information acquisition electronic glasses; Screen blinkpunkt detecting unit, obtain the human eye blinkpunkt on screen according to the human eye blinkpunkt on electronic glasses, glasses position and attitude information.

Alternatively, the positional information that described human eye feature information is pupil center and corneal reflection point, glasses blinkpunkt detecting unit comprises: computing unit goes out the human eye blinkpunkt of people's an eye line on electronic glasses according to the positional information calculation of pupil center and corneal reflection point.

Alternatively, the human eye motion detection unit be used for detecting close one's eyes, watch attentively, at least one human eye action in twitching of the eyelid, smooth motion.

Alternatively, control module is controlled the operation of display under different patterns according to the human eye blinkpunkt on human eye action and screen.

Alternatively, when display image screen, control module is controlled display according to the human eye blinkpunkt on human eye action and screen and is proceeded as follows: when human eye moves as watching attentively, and, when the human eye blinkpunkt on screen is positioned at the screen zone line, near the image be positioned at the human eye blinkpunkt on screen is amplified and shown; In the situation that the image of screen display is moving image, when the human eye action is level and smooth the tracking, according to the human eye blinkpunkt on the screen in level and smooth tracing process, determine the object that human eye is followed the trail of, the object that human eye is followed the trail of is illustrated in the central authorities of screen; When human eye action, for watching attentively, and the human eye blinkpunkt on screen is positioned at screen border when regional, and image is moved towards the direction away from this border.

Alternatively, when the screen display user interface, control module is controlled display according to the human eye blinkpunkt on human eye action and screen and proceeded as follows: the human eye blinkpunkt on screen is positioned at a functional area, and then human eye moves when closing one's eyes, and moves the function corresponding with this functional area; When human eye action is smooth motion, and the human eye blinkpunkt on screen is switched to this another zone by focus while from a functional area, moving to another functional area.

Alternatively, the equipment of described control display also comprises: viewing angle detecting unit, the viewing angle while according to human eye feature information, glasses position and attitude information detection, watching screen; Degree of depth adjustment unit, according to the degree of depth of viewing angle adjustment screen.

Alternatively, when viewing angle surpasses the predetermined angular threshold value, degree of depth adjustment unit is turned the degree of depth near the zone of electronic glasses on screen down, and the degree of depth away from the zone of electronic glasses on screen is tuned up, and makes viewing angle be less than or equal to the angle predetermined threshold.

A third aspect of the present invention provides a kind of method of utilizing the electronic glasses Remote Control Indicator, comprising: take human eye by electronic glasses and obtain eye image; Extract human eye feature information from eye image; Glasses position and the attitude information of detected electrons glasses; The characteristic information of the eye image that transmission is taken and the glasses position of electronic glasses and attitude information are to display.

A fourth aspect of the present invention provides a kind of equipment that utilizes the electronic glasses Remote Control Indicator, comprising: the eye image acquiring unit obtains eye image by taking human eye; Feature extraction unit is extracted human eye feature information from eye image; The position and attitude detecting unit, glasses position and the attitude information of detected electrons glasses; Transmitting element, the characteristic information of the eye image that transmission is taken and the glasses position of electronic glasses and attitude information are to display.

A fifth aspect of the present invention provides a kind of method of controlling display, comprising: from electronic glasses, receive the characteristic information of the eye image of taking and glasses position and the attitude information of electronic glasses; Detect the human eye blinkpunkt on screen according to human eye feature information, glasses position and attitude information; Detect the human eye action according to the situation of change of the human eye blinkpunkt on human eye feature information and/or screen; Control the operation of display according to the human eye blinkpunkt on human eye action and screen.

Alternatively, the positional information that described human eye feature information is pupil center and corneal reflection point comprises according to the step of the human eye blinkpunkt on human eye feature information acquisition electronic glasses: according to the positional information calculation of pupil center and corneal reflection point, go out the blinkpunkt of people's an eye line on electronic glasses.

Alternatively, the step of adjusting the degree of depth of screen according to viewing angle comprises: when viewing angle surpasses the predetermined angular threshold value, degree of depth adjustment unit is turned the degree of depth in the zone of close electronic glasses on screen down, the degree of depth away from the zone of electronic glasses on screen is tuned up, make viewing angle be less than or equal to the angle predetermined threshold.

A sixth aspect of the present invention provides a kind of equipment of controlling display, comprising: receiving element receives the characteristic information of the eye image of taking and glasses position and the attitude information of electronic glasses from electronic glasses; The blinkpunkt detecting unit, detect the human eye blinkpunkt on screen according to human eye feature information, glasses position and attitude information; The human eye motion detection unit, detect the human eye action according to the situation of change of the human eye blinkpunkt on human eye feature information and/or screen; Control module, control the operation of display according to the human eye blinkpunkt on human eye action and screen.

Alternatively, when viewing angle surpasses the predetermined angular threshold value, degree of depth adjustment unit is turned the degree of depth near the zone of electronic glasses on screen down, and the degree of depth away from the zone of electronic glasses on screen is tuned up, and makes viewing angle be less than or equal to the predetermined angular threshold value.

A seventh aspect of the present invention provides a kind of method of utilizing the electronic glasses Remote Control Indicator, comprising: take human eye by electronic glasses and obtain eye image; Extract human eye feature information from eye image; Glasses position and the attitude information of detected electrons glasses; Detect the human eye blinkpunkt on screen according to human eye feature information, glasses position and attitude information; Detect the human eye action according to the situation of change of the human eye blinkpunkt on human eye feature information and/or screen; To send to display for the human eye action of the operation of controlling display and the human eye blinkpunkt on screen.

A eighth aspect of the present invention provides a kind of equipment that utilizes the electronic glasses Remote Control Indicator, comprising: the eye image acquiring unit obtains eye image by taking human eye; Feature extraction unit is extracted human eye feature information from eye image; The position and attitude detecting unit, glasses position and the attitude information of detected electrons glasses; The blinkpunkt detecting unit, detect the human eye blinkpunkt on screen according to human eye feature information, glasses position and attitude information; The human eye motion detection unit, detect the human eye action according to the situation of change of the human eye blinkpunkt on human eye feature information and/or screen; Transmitting element, will send to display for the human eye action of the operation of controlling display and the human eye blinkpunkt on screen.

Alternatively, the blinkpunkt detecting unit be used for detecting close one's eyes, watch attentively, at least one human eye action in twitching of the eyelid, smooth motion.

A ninth aspect of the present invention provides a kind of method of controlling display, comprising: from electronic glasses, receive the human eye action of the operation of controlling display and the human eye blinkpunkt screen; Control the operation of display according to the human eye blinkpunkt on human eye action and screen.

A tenth aspect of the present invention provides a kind of equipment of controlling display, comprising: receiving element receives the human eye action of the operation of controlling display and the human eye blinkpunkt screen from electronic glasses; Control module, control the operation of display according to the human eye blinkpunkt on human eye action and screen.

According to the method and apparatus of control display of the present invention, can control display according to eye image, user's both hands are freed from telechiric device.

The accompanying drawing explanation

By the description of embodiment being carried out below in conjunction with accompanying drawing, these and/or other aspect of the present invention and advantage will become clear and be easier to and understand, wherein:

Fig. 1 illustrates the process flow diagram according to the method for the control display of the first exemplary embodiment of the present invention.

Fig. 2 illustrates the block diagram according to the equipment of the control display of the second exemplary embodiment of the present invention.

Fig. 3 illustrates the process flow diagram according to the method for utilizing the electronic glasses Remote Control Indicator of the 3rd exemplary embodiment of the present invention.

Fig. 4 illustrates the block diagram according to the equipment that utilizes the electronic glasses Remote Control Indicator of the 4th exemplary embodiment of the present invention.

Fig. 5 illustrates the process flow diagram according to the method for the control display of the 5th exemplary embodiment of the present invention.

Fig. 6 illustrates the block diagram according to the equipment of the control display of the 6th exemplary embodiment of the present invention.

Fig. 7 illustrates the process flow diagram according to the method for utilizing the electronic glasses Remote Control Indicator of the 7th exemplary embodiment of the present invention.

Fig. 8 illustrates the block diagram according to the equipment that utilizes the electronic glasses Remote Control Indicator of the 8th exemplary embodiment of the present invention.

Fig. 9 illustrates the process flow diagram according to the method for the control display of the 9th exemplary embodiment of the present invention.

Figure 10 illustrates the block diagram according to the equipment of the control display of the tenth exemplary embodiment of the present invention.

Embodiment

Now the embodiment of the present invention is described in detail, its example shown in the accompanying drawings.Below with reference to the accompanying drawings embodiment is described to explain the present invention.

In step 101, obtain the eye image of taking by electronic glasses.The user wears electronic glasses when watching screen, takes user's eyes acquisition user's eye image by the image capturing device on electronic glasses.

In step 102, extract human eye feature information from eye image.Human eye feature information is the information that can characterize people's an eye line.For example, human eye feature information can be the positional information of pupil center and corneal reflection point, according to the positional information of pupil center and corneal reflection point, can obtain people's an eye line information.

In step 103, glasses position and the attitude information of detected electrons glasses.The glasses position of electronic glasses and attitude information can be used predetermined coordinate system to be expressed.

In step 104, according to human eye feature information, glasses position and attitude information, detect the human eye blinkpunkt on screen.At first according to the human eye blinkpunkt on human eye feature information acquisition electronic glasses.Then obtain the human eye blinkpunkt on screen according to the human eye blinkpunkt on electronic glasses, glasses position and attitude information.

Specifically, can realize according to the people's an eye line on human eye feature acquisition of information electronic glasses coordinate system by existing Eye Tracking Technique, then obtain the human eye blinkpunkt on electronic glasses according to the people's an eye line on the electronic glasses coordinate system.Then, according to position and the attitude information of the human eye blinkpunkt on electronic glasses and electronic glasses, carry out corresponding coordinate transform, obtain the human eye blinkpunkt on screen.

In one embodiment, the positional information that described human eye feature information is pupil center and corneal reflection point, go out the human eye blinkpunkt of people's an eye line on electronic glasses according to the positional information calculation of pupil center and corneal reflection point.That is, use pupil-corneal reflection vector method to analyze the human eye characteristic information and obtain the people's an eye line on the electronic glasses coordinate system, according to the people's an eye line on the electronic glasses coordinate system, obtain the human eye blinkpunkt on electronic glasses.

In step 105, according to the situation of change of the human eye blinkpunkt on human eye feature information and/or screen, detect the human eye action.Comprise close one's eyes, watch attentively, the human eye actions such as twitching of the eyelid, smooth motion.In the time of can't from human eye feature information, obtaining corneal reflection point within first schedule time, the human eye motion detection is for closing one's eyes.When the human eye blinkpunkt on screen remains unchanged within second schedule time, the human eye motion detection is for watching attentively; When the moving range within the 3rd schedule time of the human eye blinkpunkt on screen surpasses predetermined threshold, the human eye motion detection is twitching of the eyelid; When the human eye blinkpunkt on screen moves at a predetermined velocity, the human eye motion detection is smooth motion.Above predetermined value can be preset or be arranged by the user.

In step 106, control the operation of display according to the human eye blinkpunkt on human eye action and screen.Specifically, can by human eye move and screen on the various combination of human eye blinkpunkt control the various operations of display.

For example, the position at cursor place when the human eye blinkpunkt on screen is equivalent to use the mouse control display, or the position that while controlling display, telepilot is chosen that uses a teleswitch, the human eye action is equivalent to click the mouse or presses the button of telepilot.For example, action be equivalent to the to click the mouse left button or press the confirmation button of telepilot of closing one's eyes.

Preferably, control the operation of display according to the human eye blinkpunkt on human eye action and screen under different patterns.

Under watching mode, when human eye action, for watching attentively, and the human eye blinkpunkt on screen amplified and shown the image that is positioned at the blinkpunkt near zone while being positioned at the screen zone line, facilitates the user to be clear that the trickle image of oneself paying close attention to.The size of the blinkpunkt near zone amplified here, can be preset or be arranged by the user.In the situation that the image of screen display is moving image, when the human eye action is level and smooth the tracking, determine according to the human eye blinkpunkt on the screen in level and smooth tracing process the object that human eye is followed the trail of, extract the information of this object from TV program stream, this object is followed the trail of, followed the trail of object is illustrated in to the central authorities of screen.When human eye action, for watching attentively, and the human eye blinkpunkt on screen is positioned at screen border when regional, and image is moved towards the direction away from this border.For example: when human eye action, for watching attentively, and the human eye blinkpunkt on screen is positioned at the screen left margin when regional, and image is moved to the right, and the picture showing that left side is not represented out.

Under user interface mode, the image of screen is divided into several zones, the specific function of each Regional Representative.Human eye blinkpunkt on screen is positioned at a functional area, and then human eye action when closing one's eyes, moves the function corresponding with this functional area.

For example: when the user watches the page-turning function region attentively, while then closing one's eyes, the operation page-turning function.When human eye action is smooth motion, and the human eye blinkpunkt on screen is switched to this another zone (that is, from choosing a zone to become, choosing another zone) by focus while from a functional area, moving to another functional area.

In addition, should be appreciated that, function can also be to call function that the built-in function (for example, brightness regulation, mode tuning etc.) of the built-in function of various application software, application software, display or User Defined arrange etc.

In addition, according to human eye feature information, glasses position and attitude information, detect the viewing angle while watching screen, then according to viewing angle, adjust the degree of depth of screen.When viewing angle surpasses the predetermined angular threshold value; the degree of depth in the zone of close electronic glasses on screen is turned down; the degree of depth away from the zone of electronic glasses on screen is tuned up, make viewing angle be less than or equal to the predetermined angular threshold value, thereby play the purpose of protection eyes.

Should be appreciated that, step 101-106 can be all realized by the electronic glasses telechiric device of electronic glasses (or have) or display; Perhaps part is realized by the electronic glasses telechiric device of electronic glasses (or have), and other parts are realized by display.

As shown in Figure 2, according to the equipment 200 of control display of the present invention, comprise: eye image acquiring unit 201, feature extraction unit 202, position and attitude detecting unit 203, blinkpunkt detecting unit 204, human eye motion detection unit 205, control module 206.

Eye image acquiring unit 201 is for obtaining eye image by utilizing electronic glasses to take human eye.Electronic glasses can have the CMOS(complementary metal oxide semiconductor (CMOS)) sensor, CCD(charge-coupled image sensor) the various filming apparatus of sensor.

Feature extraction unit 202 is extracted human eye feature information for the eye image from obtaining.Human eye feature information is the information that can characterize people's an eye line.For example, human eye feature information can be the positional information of pupil center and corneal reflection point, according to the positional information of pupil center and corneal reflection point, can obtain people's an eye line information.

Position and attitude detecting unit 203 is for detection of glasses position and the attitude information of electronic glasses.Position and attitude detecting unit 203 can be various position and attitude pick-up units.

Blinkpunkt detecting unit 204 is for detecting the human eye blinkpunkt on screen according to human eye feature information, glasses position and attitude information.

Blinkpunkt detecting unit 204 comprises glasses blinkpunkt detecting unit (not shown) and screen blinkpunkt detecting unit (not shown).Glasses blinkpunkt detecting unit is used for according to the human eye blinkpunkt on human eye feature information acquisition electronic glasses.Specifically, glasses blinkpunkt detecting unit can be realized according to the people's an eye line on human eye feature acquisition of information electronic glasses coordinate system by existing Eye Tracking Technique, then obtain the human eye blinkpunkt on electronic glasses according to the people's an eye line on the electronic glasses coordinate system.Screen blinkpunkt detecting unit, for according to position and the attitude information of the human eye blinkpunkt on electronic glasses and electronic glasses, carrying out corresponding coordinate transform, obtains the human eye blinkpunkt on screen.

In one embodiment, the positional information that described human eye feature information is pupil center and corneal reflection point, glasses blinkpunkt detecting unit also comprises the computing unit (not shown).Computing unit goes out the human eye blinkpunkt of people's an eye line on electronic glasses according to the positional information calculation of pupil center and corneal reflection point.That is, use pupil-corneal reflection vector method to analyze the human eye characteristic information and obtain the people's an eye line on the electronic glasses coordinate system, according to the people's an eye line on the electronic glasses coordinate system, obtain the human eye blinkpunkt on electronic glasses.

Human eye motion detection unit 205 detects the human eye action for the situation of change of the human eye blinkpunkt according on human eye feature information and/or screen.Comprise close one's eyes, watch attentively, the human eye actions such as twitching of the eyelid, smooth motion.In the time of can't from human eye feature information, obtaining corneal reflection point within first schedule time, the human eye motion detection is for closing one's eyes.When the human eye blinkpunkt on screen remains unchanged within second schedule time, the human eye motion detection is for watching attentively; When the moving range within the 3rd schedule time of the human eye blinkpunkt on screen surpasses predetermined threshold, the human eye motion detection is twitching of the eyelid; When the human eye blinkpunkt on screen moves at a predetermined velocity, the human eye motion detection is smooth motion.Above predetermined value can be preset or be arranged by the user.

Control module 206 is controlled the operation of display for the human eye blinkpunkt according on human eye action and screen.Specifically, control module 206 can by human eye move and screen on the various combination of human eye blinkpunkt control the various operations of display.

Preferably, control module 206 is controlled the operation of display under different patterns according to the human eye blinkpunkt on human eye action and screen.

Under watching mode, when human eye action, for watching attentively, and the human eye blinkpunkt on screen amplified and shown the image that is positioned at the blinkpunkt near zone while being positioned at the screen zone line, facilitates the user to be clear that the trickle image of oneself paying close attention to.The size of the blinkpunkt near zone amplified can be preset or be arranged by the user.In the situation that the image of screen display is moving image, when the human eye action is level and smooth the tracking, determine according to the human eye blinkpunkt on the screen in level and smooth tracing process the object that human eye is followed the trail of, extract the information of this object from TV program stream, this object is followed the trail of, followed the trail of object is illustrated in to the central authorities of screen.When human eye action, for watching attentively, and the human eye blinkpunkt on screen is positioned at screen border when regional, and image is moved towards the direction away from this border.For example: when human eye action, for watching attentively, and the human eye blinkpunkt on screen is positioned at the screen left margin when regional, and image is moved to the right, and the picture showing that left side is not represented out.

The equipment 200 of controlling display also comprises viewing angle detecting unit (not shown) and degree of depth adjustment unit (not shown).Viewing angle when the viewing angle detecting unit is watched screen according to human eye feature information, glasses position and attitude information detection, degree of depth adjustment unit is adjusted the degree of depth of screen according to viewing angle.When viewing angle surpasses the predetermined angular threshold value, the degree of depth near the zone of electronic glasses on screen is turned down, the degree of depth away from the zone of electronic glasses on screen is tuned up, make viewing angle be less than or equal to the predetermined angular threshold value, play the purpose of protection eyes.

Should be appreciated that, unit 101-106 can all be arranged in the electronic glasses telechiric device of electronic glasses (or have) or display; Perhaps part is arranged in the electronic glasses telechiric device of electronic glasses (or have), and other parts are arranged in display.

In step 301, obtain the eye image of taking by electronic glasses.The user wears electronic glasses when watching screen, takes user's eyes acquisition user's eye image by the image capturing device on electronic glasses.

In step 302, extract human eye feature information from eye image.Human eye feature information is the information that can characterize people's an eye line.For example, human eye feature information can be the positional information of pupil center and corneal reflection point, according to the positional information of pupil center and corneal reflection point, can obtain people's an eye line information.

In step 303, glasses position and the attitude information of detected electrons glasses.The glasses position of electronic glasses and attitude information can be used predetermined coordinate system to be expressed.

In step 304, the characteristic information of the eye image that transmission is taken and the glasses position of electronic glasses and attitude information are to display.

Should be appreciated that, above step can be realized by electronic glasses or the telechiric device with electronic glasses.

As shown in Figure 4, the equipment 400 of electronic glasses Remote Control Indicator that utilizes according to the present invention comprises: eye image acquiring unit 401, feature extraction unit 402, position and attitude detecting unit 403, transmitting element 404.

Eye image acquiring unit 401 is for obtaining eye image by taking human eye.Eye image acquiring unit 401 can be the CMOS(complementary metal oxide semiconductor (CMOS)) sensor, CCD(charge-coupled image sensor) the various filming apparatus of sensor.

Feature extraction unit 402 is extracted human eye feature information for the eye image from obtaining.Human eye feature information is the information that can characterize people's an eye line.For example, human eye feature information can be the positional information of pupil center and corneal reflection point, according to the positional information of pupil center and corneal reflection point, can obtain people's an eye line information.

Position and attitude detecting unit 403 is for detection of glasses position and the attitude information of electronic glasses.Position and attitude detecting unit 403 can be various position and attitude pick-up units.

Transmitting element 404 for the glasses position of the characteristic information of the eye image that sends shooting and electronic glasses and attitude information to display.Transmitting element 404 can be various wireless data transmission devices.

Should be appreciated that, equipment 400 can be implemented as electronic glasses or has the telechiric device of electronic glasses.

In step 501, from electronic glasses, receive the characteristic information of the eye image of taking and glasses position and the attitude information of electronic glasses.

In step 502, according to human eye feature information, glasses position and attitude information, detect the human eye blinkpunkt on screen.At first according to the human eye blinkpunkt on human eye feature information acquisition electronic glasses.Then obtain the human eye blinkpunkt on screen according to the human eye blinkpunkt on electronic glasses, glasses position and attitude information.Specifically, can realize according to the people's an eye line on human eye feature acquisition of information electronic glasses coordinate system by existing Eye Tracking Technique, then obtain the human eye blinkpunkt on electronic glasses according to the people's an eye line on the electronic glasses coordinate system.Then, according to position and the attitude information of the human eye blinkpunkt on electronic glasses and electronic glasses, carry out corresponding coordinate transform, obtain the human eye blinkpunkt on screen.

In step 503, according to the situation of change of the human eye blinkpunkt on human eye feature information and/or screen, detect the human eye action.Comprise close one's eyes, watch attentively, the human eye actions such as twitching of the eyelid, smooth motion.In the time of can't from human eye feature information, obtaining corneal reflection point within first schedule time, the human eye motion detection is for closing one's eyes.When the human eye blinkpunkt on screen remains unchanged within second schedule time, the human eye motion detection is for watching attentively; When the moving range within the 3rd schedule time of the human eye blinkpunkt on screen surpasses predetermined threshold, the human eye motion detection is twitching of the eyelid; When the human eye blinkpunkt on screen moves at a predetermined velocity, the human eye motion detection is smooth motion.Above predetermined value can be preset or be arranged by the user.

In step 504, control the operation of display according to the human eye blinkpunkt on human eye action and screen.Specifically, can by human eye move and screen on the various combination of human eye blinkpunkt control the various operations of display.

Under watching mode, when human eye action, for watching attentively, and the human eye blinkpunkt on screen amplified and shown the image that is positioned at the blinkpunkt near zone while being positioned at the screen zone line, facilitates the user to be clear that the trickle image of oneself paying close attention to.The size of the blinkpunkt near zone wherein, amplified can be preset or be arranged by the user.In the situation that the image of screen display is moving image, when the human eye action is level and smooth the tracking, determine according to the human eye blinkpunkt on the screen in level and smooth tracing process the object that human eye is followed the trail of, extract the information of this object from TV program stream, this object is followed the trail of, followed the trail of object is illustrated in to the central authorities of screen.When human eye action, for watching attentively, and the human eye blinkpunkt on screen is positioned at screen border when regional, and image is moved towards the direction away from this border.For example: when human eye action, for watching attentively, and the human eye blinkpunkt on screen is positioned at the screen left margin when regional, and image is moved to the right, and the picture showing that left side is not represented out.

Should be appreciated that, above step can be realized by display.

As shown in Figure 6, according to the equipment 600 of control display of the present invention, comprise: receiving element 601, blinkpunkt detecting unit 602, human eye motion detection unit 603, control module 604.

Receiving element 601 is for receiving the characteristic information of the eye image of taking and glasses position and the attitude information of electronic glasses from electronic glasses.

Blinkpunkt detecting unit 602 is for detecting the human eye blinkpunkt on screen according to human eye feature information, glasses position and attitude information.

Blinkpunkt detecting unit 602 comprises glasses blinkpunkt detecting unit (not shown) and screen blinkpunkt detecting unit (not shown).Glasses blinkpunkt detecting unit is used for according to the human eye blinkpunkt on human eye feature information acquisition electronic glasses.Specifically, glasses blinkpunkt detecting unit can be realized according to the people's an eye line on human eye feature acquisition of information electronic glasses coordinate system by existing Eye Tracking Technique, then obtain the human eye blinkpunkt on electronic glasses according to the people's an eye line on the electronic glasses coordinate system.Screen blinkpunkt detecting unit, for according to position and the attitude information of the human eye blinkpunkt on electronic glasses and electronic glasses, carrying out corresponding coordinate transform, obtains the human eye blinkpunkt on screen.

Human eye motion detection unit 603 detects the human eye action for the situation of change of the human eye blinkpunkt according on human eye feature information and/or screen.Comprise close one's eyes, watch attentively, the human eye actions such as twitching of the eyelid, smooth motion.In the time of can't from human eye feature information, obtaining corneal reflection point within first schedule time, the human eye motion detection is for closing one's eyes.When the human eye blinkpunkt on screen remains unchanged within second schedule time, the human eye motion detection is for watching attentively; When the moving range within the 3rd schedule time of the human eye blinkpunkt on screen surpasses predetermined threshold, the human eye motion detection is twitching of the eyelid; When the human eye blinkpunkt on screen moves at a predetermined velocity, the human eye motion detection is smooth motion.Above predetermined value can be preset or be arranged by the user.

Control module 604 is controlled the operation of display for the human eye blinkpunkt according on human eye action and screen.Specifically, can by human eye move and screen on the various combination of human eye blinkpunkt control the various operations of display.

Preferably, control module 604 is controlled the operation of display under different patterns according to the human eye blinkpunkt on human eye action and screen.

The equipment 600 of controlling display also comprises viewing angle detecting unit (not shown) and degree of depth adjustment unit (not shown).Viewing angle when the viewing angle detecting unit is watched screen according to human eye feature information, glasses position and attitude information detection, degree of depth adjustment unit is adjusted the degree of depth of screen according to viewing angle.When viewing angle surpasses the predetermined angular threshold value, the degree of depth near the zone of electronic glasses on screen is turned down, the degree of depth away from the zone of electronic glasses on screen is tuned up, make viewing angle be less than or equal to the predetermined angular threshold value, play the purpose of protection eyes.

Should be appreciated that, equipment 600 can be arranged in display.

In step 701, obtain the eye image of taking by electronic glasses.The user wears electronic glasses when watching screen, takes user's eyes acquisition user's eye image by the image capturing device on electronic glasses.

In step 702, extract human eye feature information from eye image.Human eye feature information is the information that can characterize people's an eye line.For example, human eye feature information can be the positional information of pupil center and corneal reflection point, according to the positional information of pupil center and corneal reflection point, can obtain people's an eye line information.

In step 703, glasses position and the attitude information of detected electrons glasses.The glasses position of electronic glasses and attitude information can be used predetermined coordinate system to be expressed.

In step 704, according to human eye feature information, glasses position and attitude information, detect the human eye blinkpunkt on screen.At first according to the human eye blinkpunkt on human eye feature information acquisition electronic glasses.Then obtain the human eye blinkpunkt on screen according to the human eye blinkpunkt on electronic glasses, glasses position and attitude information.Specifically, can realize according to the people's an eye line on human eye feature acquisition of information electronic glasses coordinate system by existing Eye Tracking Technique, then obtain the human eye blinkpunkt on electronic glasses according to the people's an eye line on the electronic glasses coordinate system.Then, according to position and the attitude information of the human eye blinkpunkt on electronic glasses and electronic glasses, carry out corresponding coordinate transform, obtain the human eye blinkpunkt on screen.

In step 705, according to the situation of change of the human eye blinkpunkt on human eye feature information and/or screen, detect the human eye action.Comprise close one's eyes, watch attentively, the human eye actions such as twitching of the eyelid, smooth motion.In the time of can't from human eye feature information, obtaining corneal reflection point within first schedule time, the human eye motion detection is for closing one's eyes.When the human eye blinkpunkt on screen remains unchanged within second schedule time, the human eye motion detection is for watching attentively; When the moving range within the 3rd schedule time of the human eye blinkpunkt on screen surpasses predetermined threshold, the human eye motion detection is twitching of the eyelid; When the human eye blinkpunkt on screen moves at a predetermined velocity, the human eye motion detection is smooth motion.Above predetermined value can be preset or be arranged by the user.

In step 706, the characteristic information of the eye image that transmission is taken and the glasses position of electronic glasses and attitude information are to display.

As shown in Figure 8, according to the equipment 800 of control display of the present invention, comprise: eye image acquiring unit 801, feature extraction unit 802, position and attitude detecting unit 803, blinkpunkt detecting unit 804, human eye motion detection unit 805, transmitting element 806.

Eye image acquiring unit 801 is for obtaining eye image by taking human eye.Eye image acquiring unit 801 can be the CMOS(complementary metal oxide semiconductor (CMOS)) sensor, CCD(charge-coupled image sensor) the various filming apparatus of sensor.

Feature extraction unit 802 is extracted human eye feature information for the eye image from obtaining.Human eye feature information is the information that can characterize people's an eye line.For example, human eye feature information can be the positional information of pupil center and corneal reflection point, according to the positional information of pupil center and corneal reflection point, can obtain people's an eye line information.

Position and attitude detecting unit 803 is for detection of glasses position and the attitude information of electronic glasses.Position and attitude detecting unit 803 can be various position and attitude pick-up units.

Blinkpunkt detecting unit 804 is for detecting the human eye blinkpunkt on screen according to human eye feature information, glasses position and attitude information.

Blinkpunkt detecting unit 804 comprises glasses blinkpunkt detecting unit (not shown) and screen blinkpunkt detecting unit (not shown).Glasses blinkpunkt detecting unit is used for according to the human eye blinkpunkt on human eye feature information acquisition electronic glasses.Specifically, glasses blinkpunkt detecting unit can be realized according to the people's an eye line on human eye feature acquisition of information electronic glasses coordinate system by existing Eye Tracking Technique, then obtain the human eye blinkpunkt on electronic glasses according to the people's an eye line on the electronic glasses coordinate system.Screen blinkpunkt detecting unit, for according to position and the attitude information of the human eye blinkpunkt on electronic glasses and electronic glasses, carrying out corresponding coordinate transform, obtains the human eye blinkpunkt on screen.

Human eye motion detection unit 805 detects the human eye action for the situation of change of the human eye blinkpunkt according on human eye feature information and/or screen.Comprise close one's eyes, watch attentively, the human eye actions such as twitching of the eyelid, smooth motion.In the time of can't from human eye feature information, obtaining corneal reflection point within first schedule time, the human eye motion detection is for closing one's eyes.When the human eye blinkpunkt on screen remains unchanged within second schedule time, the human eye motion detection is for watching attentively; When the moving range within the 3rd schedule time of the human eye blinkpunkt on screen surpasses predetermined threshold, the human eye motion detection is twitching of the eyelid; When the human eye blinkpunkt on screen moves at a predetermined velocity, the human eye motion detection is smooth motion.Above predetermined value can be preset or be arranged by the user.

Transmitting element 806 for the glasses position of the characteristic information of the eye image that sends shooting and electronic glasses and attitude information to display.Transmitting element 806 can be various wireless data transmission devices.

Should be appreciated that, equipment 800 can be implemented as electronic glasses or has the telechiric device of electronic glasses.

In step 901, from electronic glasses, receive the characteristic information of the eye image of taking and glasses position and the attitude information of electronic glasses.

In step 902, control the operation of display according to the human eye blinkpunkt on human eye action and screen.Specifically, can by human eye move and screen on the various combination of human eye blinkpunkt control the various operations of display.

Should be appreciated that, above step can be realized by display.

As shown in figure 10, according to the equipment 1000 of control display of the present invention, comprise: receiving element 1001, control module 1002.

Receiving element 1001 is for receiving the characteristic information of the eye image of taking and glasses position and the attitude information of electronic glasses from electronic glasses.

Control module 1002 is controlled the operation of display for the human eye blinkpunkt according on human eye action and screen.Specifically, can by human eye move and screen on the various combination of human eye blinkpunkt control the various operations of display.

Preferably, control module 1002 is controlled the operation of display under different patterns according to the human eye blinkpunkt on human eye action and screen.

Should be appreciated that, equipment 1000 can be arranged in display.

In addition, according to the said method of exemplary embodiment of the present invention, may be implemented as computer program, thereby, when this program of operation, realize said method.Can be implemented nextport hardware component NextPort according to the unit in the equipment of the control display of exemplary embodiment of the present invention.Those skilled in the art, according to the performed processing of unit limited, can for example use field programmable gate array (FPGA) or special IC (ASIC) to realize unit.

Although meaned and described some embodiments of the present invention, having it should be appreciated by those skilled in the art that and can modify to these embodiment in the situation that do not break away from principle of the present invention and the spirit that is limited its scope by claim and equivalent thereof.

Claims

1. a method of controlling display comprises:

Take human eye by electronic glasses and obtain eye image;

Extract human eye feature information from eye image;

Glasses position and the attitude information of detected electrons glasses;

Detect the human eye blinkpunkt on screen according to human eye feature information, glasses position and attitude information;

Detect the human eye action according to the situation of change of the human eye blinkpunkt on human eye feature information and/or screen;

Control the operation of display according to the human eye blinkpunkt on human eye action and screen.

2. the method for claim 1, wherein according to the step of the human eye blinkpunkt on human eye feature information, glasses position and attitude information detection screen, comprise:

According to the human eye blinkpunkt on human eye feature information acquisition electronic glasses;

Obtain the human eye blinkpunkt on screen according to the human eye blinkpunkt on electronic glasses, glasses position and attitude information.

3. method as claimed in claim 2, wherein, the positional information that described human eye feature information is pupil center and corneal reflection point comprises according to the step of the human eye blinkpunkt on human eye feature information acquisition electronic glasses:

Go out the human eye blinkpunkt of people's an eye line on electronic glasses according to the positional information calculation of pupil center and corneal reflection point.

The method of claim 1, wherein the action of described human eye comprise close one's eyes, watch attentively, at least one in twitching of the eyelid, smooth motion.

5. method as claimed in claim 4, wherein,

In the time of can't from human eye feature information, obtaining corneal reflection point within first schedule time, the human eye motion detection is for closing one's eyes;

When the human eye blinkpunkt on screen remains unchanged within second schedule time, the human eye motion detection is for watching attentively;

When the moving range within the 3rd schedule time of the human eye blinkpunkt on screen surpasses predetermined threshold, the human eye motion detection is twitching of the eyelid;

When the human eye blinkpunkt on screen moves at a predetermined velocity, the human eye motion detection is smooth motion.

6. the method for claim 1, wherein under different patterns, according to the human eye blinkpunkt on human eye action and screen, control the operation of display.

7. the method for claim 1, wherein, the step of controlling the operation of display according to the human eye blinkpunkt on human eye action and screen comprises: when display image screen, control display according to the human eye blinkpunkt on human eye action and screen and proceed as follows:

When human eye action, for watching attentively, and the human eye blinkpunkt on screen is while being positioned at the screen zone line, and near the image be positioned at the human eye blinkpunkt on screen is amplified and shows;

In the situation that the image of screen display is moving image, when the human eye action is level and smooth the tracking, according to the human eye blinkpunkt on the screen in level and smooth tracing process, determine the object that human eye is followed the trail of, the object that human eye is followed the trail of is illustrated in the central authorities of screen;

When human eye action, for watching attentively, and the human eye blinkpunkt on screen is positioned at screen border when regional, and image is moved towards the direction away from this border.

8. the method for claim 1, wherein, the step of controlling the operation of display according to the human eye blinkpunkt on human eye action and screen comprises: when the screen display user interface, control display according to the human eye blinkpunkt on human eye action and screen and proceed as follows:

Human eye blinkpunkt on screen is positioned at a functional area, and then human eye action when closing one's eyes, moves the function corresponding with this functional area;

When human eye action is smooth motion, and the human eye blinkpunkt on screen is switched to this another zone by focus while from a functional area, moving to another functional area.

9. the method for claim 1 also comprises:

Viewing angle while according to human eye feature information, glasses position and attitude information detection, watching screen;

Adjust the degree of depth of screen according to viewing angle.

10. method as claimed in claim 9, wherein, the step of adjusting the degree of depth of screen according to viewing angle comprises:

When viewing angle surpasses the predetermined angular threshold value, the degree of depth near the zone of electronic glasses on screen is turned down, the degree of depth away from the zone of electronic glasses on screen is tuned up, make viewing angle be less than or equal to the predetermined angular threshold value.

11. an equipment of controlling display comprises:

The eye image acquiring unit, take human eye by electronic glasses and obtain eye image;

Feature extraction unit is extracted human eye feature information from eye image;

The position and attitude detecting unit, glasses position and the attitude information of detected electrons glasses;

The blinkpunkt detecting unit, detect the human eye blinkpunkt on screen according to human eye feature information, glasses position and attitude information;

The human eye motion detection unit, detect the human eye action according to the situation of change of the human eye blinkpunkt on human eye feature information and/or screen;

Control module, control the operation of display according to the human eye blinkpunkt on human eye action and screen.

12. equipment as claimed in claim 11, wherein, the blinkpunkt detecting unit comprises:

Glasses blinkpunkt detecting unit, according to the human eye blinkpunkt on human eye feature information acquisition electronic glasses;

Screen blinkpunkt detecting unit, obtain the human eye blinkpunkt on screen according to the human eye blinkpunkt on electronic glasses, glasses position and attitude information.

13. equipment as claimed in claim 12, wherein, the positional information that described human eye feature information is pupil center and corneal reflection point, glasses blinkpunkt detecting unit comprises:

Computing unit, go out the human eye blinkpunkt of people's an eye line on electronic glasses according to the positional information calculation of pupil center and corneal reflection point.

14. equipment as claimed in claim 11, wherein, the human eye motion detection unit is used for detecting and closes one's eyes, watches attentively, at least one human eye action in twitching of the eyelid, smooth motion.

15. equipment as claimed in claim 14, wherein,

16. equipment as claimed in claim 11, wherein, control module is controlled the operation of display under different patterns according to the human eye blinkpunkt on human eye action and screen.

17. equipment as claimed in claim 11, wherein, when display image screen, control module is controlled display according to the human eye blinkpunkt on human eye action and screen and is proceeded as follows:

18. equipment as claimed in claim 11, wherein, when the screen display user interface, control module is controlled display according to the human eye blinkpunkt on human eye action and screen and is proceeded as follows:

19. equipment as claimed in claim 11 also comprises:

The viewing angle detecting unit, the viewing angle while according to human eye feature information, glasses position and attitude information detection, watching screen;

Degree of depth adjustment unit, according to the degree of depth of viewing angle adjustment screen.

20. equipment as claimed in claim 19, wherein, when viewing angle surpasses the predetermined angular threshold value, degree of depth adjustment unit is turned the degree of depth in the zone of close electronic glasses on screen down, the degree of depth away from the zone of electronic glasses on screen is tuned up, make viewing angle be less than or equal to the angle predetermined threshold.

21. a method of utilizing the electronic glasses Remote Control Indicator comprises:

Take human eye by electronic glasses and obtain eye image;

Extract human eye feature information from eye image;

Glasses position and the attitude information of detected electrons glasses;

The characteristic information of the eye image that transmission is taken and the glasses position of electronic glasses and attitude information are to display.

22. an equipment that utilizes the electronic glasses Remote Control Indicator comprises:

The eye image acquiring unit, obtain eye image by taking human eye;

Transmitting element, the characteristic information of the eye image that transmission is taken and the glasses position of electronic glasses and attitude information are to display.

23. a method of controlling display comprises:

Receive the characteristic information of the eye image of taking and glasses position and the attitude information of electronic glasses from electronic glasses;

24. an equipment of controlling display comprises:

Receiving element, receive the characteristic information of the eye image of taking and glasses position and the attitude information of electronic glasses from electronic glasses;

25. a method of utilizing the electronic glasses Remote Control Indicator comprises:

Take human eye by electronic glasses and obtain eye image;

Extract human eye feature information from eye image;

Glasses position and the attitude information of detected electrons glasses;

To send to display for the human eye action of the operation of controlling display and the human eye blinkpunkt on screen.

26. an equipment that utilizes the electronic glasses Remote Control Indicator comprises:

The eye image acquiring unit, obtain eye image by taking human eye;

Transmitting element, will send to display for the human eye action of the operation of controlling display and the human eye blinkpunkt on screen.

27. a method of controlling display comprises:

Receive the human eye action of the operation of controlling display and the human eye blinkpunkt screen from electronic glasses;

28. an equipment of controlling display comprises:

Receiving element, receive the human eye action of the operation of controlling display and the human eye blinkpunkt screen from electronic glasses;