WO2023082952A1

WO2023082952A1 - Method for interacting with electronic device, and electronic device

Info

Publication number: WO2023082952A1
Application number: PCT/CN2022/125889
Authority: WO
Inventors: 刘璕
Original assignee: 华为技术有限公司
Priority date: 2021-11-10
Filing date: 2022-10-18
Publication date: 2023-05-19
Also published as: CN116107419A

Abstract

A method for interacting with an electronic device, and an electronic device, relating to the technical field of terminals. According to the method, in the operation process of the electronic device, the eye movement type of eyes of a user can be tested, and when the eye movement type is a saccade, the electronic device is controlled according to the eye movement type at the moment, and when the eye movement type is not a saccade, the user can be instructed to perform gesture control. After the user is instructed to perform gesture control, an operation gesture of the user can be obtained, and the electronic device is controlled according to the obtained operation gesture. Therefore, eye movement control and gesture control are combined on the electronic device, so that non-inductive interaction between eye movement control and gesture control can be realized, and thus the user can efficiently and conveniently control the electronic device, and the user experience is improved.

Description

A method for interacting with an electronic device and the electronic device

This application claims the priority of a Chinese patent application filed with the State Intellectual Property Office of China on November 10, 2021, with the application number 202111329181.9, and the title of the application is "A Method for Interacting with Electronic Devices and Electronic Devices", the entire content of which Incorporated in this application by reference.

technical field

The present application relates to the technical field of terminals, and in particular to a method for interacting with an electronic device and the electronic device.

Background technique

Electronic devices have become an integral part of people's lives. In terms of controlling electronic equipment, people usually use a hand-held remote control device or control buttons on the electronic equipment to control the electronic equipment, and the operation is cumbersome.

In related technologies, it is possible to detect the gaze point and movement of the user's eyes, and control the electronic device according to the preset gaze point and movement. However, during the manipulation process, the user's eyes need to frequently make deliberate movements, which is easy Cause eye fatigue. Therefore, how to realize efficient and convenient control of electronic devices is a technical problem that needs to be solved urgently.

Contents of the invention

The present application provides a method for interacting with an electronic device, an electronic device, a computer storage medium, and a computer program product, which can realize efficient and convenient control of the electronic device by combining eye movement control and gesture control, and improve user experience .

In a first aspect, the present application provides a method for interacting with an electronic device, the method including: the electronic device acquires the eye movement type of the user's eyes; when the eye movement type is determined to be the first type, the electronic device responds to the electronic The device performs control; when it is determined that the eye movement type is the second type, the electronic device instructs the user to perform gesture control; the electronic device obtains the user's gesture; and the electronic device controls the electronic device according to the obtained gesture. In this way, by combining the eye movement control and the gesture control on the electronic device, the interaction between the eye movement control and the gesture control can be realized, so that the user can efficiently and conveniently control the electronic device, and the user experience is improved. Exemplarily, the first type may be bouncing, and the second type may be non-jumping, such as head-up or smooth tracking.

According to the first aspect, instructing the user to perform gesture control specifically includes: obtaining a gaze position of the user; and instructing the user to perform gesture control within a first preset area where the gaze position is located. In this way, it is possible to instruct the user to perform gesture control in an area matching the gaze position of the user, thereby improving the accuracy of the indication and the convenience of subsequent user operations. Exemplarily, the first preset area can be understood as a surface or a point. Among them, when understood as a surface, the first preset area can be the area formed by taking the user's gaze position as the center and extending a preset distance outward; when understood as a point, the first preset area can be the user's The point where the gaze position of is, that is, the user's gaze point.

According to the first aspect, or any implementation manner of the above first aspect, instructing the user to perform gesture control specifically includes: determining the target area according to the type of the first content displayed on the electronic device; instructing the user to perform gesture control in the target area Gesture control, wherein the target area is an area on the electronic device that does not contain the first content. In this way, when instructing the user to perform gesture control, the browsing content of the user will not be affected, thereby improving user experience. Exemplarily, the first content may be text content or picture content and so on.

According to the first aspect, or any implementation of the above first aspect, instructing the user to perform gesture control specifically includes: obtaining the gaze position of the user; determining the gaze area of the user according to the gaze position; when the gaze area is the second preset area, instructs the user to perform gesture controls. In this way, the user is instructed to perform gesture control only when the user is looking at a specific area, thereby preventing the user from falsely triggering and instructing the user to perform gesture control when viewing the electronic device immersively, thereby improving user experience. For example, when the user is watching a movie, this way the user can be prevented from being falsely instructed to perform gesture controls.

According to the first aspect, or any implementation manner of the above first aspect, instructing the user to perform gesture control includes one or more of the following: controlling the electronic device to display a cursor, controlling to enlarge at least part of the content displayed by the electronic device, or, The sound component matched with the electronic device is controlled to broadcast the first voice, and the first voice is used to instruct the user to perform gesture control. The user is thus prompted dynamically or audibly for gesture control.

According to the first aspect, or any implementation of the above first aspect, instructing the user to perform gesture control includes controlling the electronic device to display a cursor, and the cursor is displayed in the first area of the display area of the electronic device; the method further includes: Determine that the eye movement type continues to be the second type, and when the user's eyes move and the user's gaze area is switched from the first area to the second area in the display area of the electronic device, control the cursor to move from the first area to the second area . In this way, during the movement of the user's eyes, when the user's gaze area changes, the cursor is controlled to move synchronously, so that the user can easily select the content he needs, and the user experience is improved.

According to the first aspect, or any implementation manner of the above first aspect, instructing the user to perform gesture control includes controlling and zooming in on at least part of the content displayed on the electronic device, and the zoomed-in content is what is displayed in the first area of the display area of the electronic device Displayed content; the method also includes: determining that the eye movement type continues to be the second type, and when the user's eyes move and the user's gaze area is switched from the first area to the second area of the electronic device, zooming in on the first area The content of the area is restored to its original state, and the content in the second area is enlarged. In this way, during the movement of the user's eyes, when the user's gaze area changes, the magnified content displayed on the electronic device is synchronously changed, so that the user can easily select the content he needs, and the user experience is improved.

According to the first aspect, or any implementation of the above first aspect, the method further includes: determining that the eye movement type is switched from the second type to the first type, stopping instructing the user to perform gesture control, and/or, the electronic device according to The first type controls the electronic device and restricts the electronic device from acquiring user's gestures.

According to the first aspect, or any implementation manner of the above first aspect, after determining that the eye movement type is the second type, it further includes: obtaining the gaze position of the user, and determining the size of the gaze area of the user according to the gaze position; According to the size of the gaze area, the size of the operation area when the user performs gesture control is determined. In this way, the user can perform gesture operations in an operation area of a size matching the size of the gaze area, thereby improving the accuracy of gesture operations and user experience.

According to the first aspect, or any implementation manner of the above first aspect, the electronic device controls the electronic device according to the first type, specifically including: the electronic device determines the gaze position of the user in real time according to the first type, and/or, The electronic device controls switching of the content displayed on the electronic device according to the first type.

In a second aspect, the present application provides a method for interacting with an electronic device, the method comprising: the electronic device obtains a user's gesture; the electronic device controls the electronic device according to the obtained gesture; the electronic device obtains the user's eye movement type; When it is determined that the user's eye movement type is the first type, the electronic device controls the electronic device according to the first type, and restricts the electronic device from controlling the electronic device according to the user's gesture; when it is determined that the user's eye movement type is the second type, the electronic device The device obtains the user's gesture; the electronic device controls the electronic device according to the obtained gesture. In this way, by combining the eye movement control and the gesture control on the electronic device, the interaction between the eye movement control and the gesture control can be realized, so that the user can efficiently and conveniently control the electronic device, and the user experience is improved. Exemplarily, the first type may be bouncing, and the second type may be non-jumping, such as head-up or smooth tracking.

According to the second aspect, or any implementation of the above second aspect, after the electronic device controls the electronic device according to the first type and restricts the electronic device from acquiring the user's gestures, the method further includes: when determining the user's eye movement When the type is switched from the first type to the second type, the electronic device instructs the user to perform gesture control; the electronic device obtains the user's gesture; and the electronic device controls the electronic device according to the obtained gesture. Therefore, eye movement control and gesture control are combined in an orderly manner, so that the eye movement control and gesture control can interact without feeling, so that the user can efficiently and conveniently control the electronic device, and the user experience is improved.

According to the second aspect, or any implementation manner of the above second aspect, restricting the electronic device from controlling the electronic device according to the user's gestures specifically includes: the electronic device does not obtain the user's gestures, or, although the electronic device obtains the user's gestures but does not Gestures are not processed. In this way, the purpose of restricting the electronic device from controlling the electronic device according to the gesture of the user is achieved.

In a third aspect, the present application provides a method for interacting with an electronic device, the method comprising: the electronic device acquires a first gesture of a user operating on a first area on the electronic device; the electronic device responds to the electronic device according to the acquired first gesture Control; the electronic device obtains the user's eye movement type; when it is determined that the user's eye movement type is the first type, the electronic device controls the electronic device according to the first type, and restricts the electronic device from obtaining the user's gesture; the electronic device continues to obtain The user's eye movement type; when it is determined that the user's eye movement type is switched from the first type to the second type, and the user's gaze position is switched from the first area on the electronic device to the second area, the electronic device instructs the user to perform gesture control The electronic device acquires a second gesture performed by the user on the second area; the electronic device controls the electronic device according to the acquired second gesture. In this way, by combining eye movement control and gesture control on the electronic device, the user can achieve the purpose of non-inductive switching when switching the control area, so that the user can efficiently and conveniently control the electronic device and improve user experience. Exemplarily, the first type may be bouncing, and the second type may be non-jumping, such as head-up or smooth tracking.

In a fourth aspect, the present application provides a method for interacting with an electronic device, the method comprising: the electronic device obtains the user's first eye movement information; the electronic device determines the first eye movement type according to the first eye movement information; the electronic device determines the first eye movement type according to The first eye movement type controls the electronic device; the electronic device obtains the user's second eye movement information; the electronic device determines the second eye movement type according to the second eye movement information; the electronic device obtains the user's gesture; the electronic device according to the acquired gesture Control electronic equipment. In this way, by combining the eye movement control and the gesture control on the electronic device, the interaction between the eye movement control and the gesture control can be realized, so that the user can efficiently and conveniently control the electronic device, and the user experience is improved. Exemplarily, the first type of eye movement may be jerky, and the second type of eye movement may be non-jiggly, such as head-up or smooth tracking. Exemplarily, the first eye movement type may also be referred to as the first type, and the second eye movement type may also be referred to as the second type.

According to the fourth aspect, or any implementation of the above fourth aspect, the method also includes: the electronic device acquires the third eye movement information of the user; the electronic device determines the first eye movement type according to the third eye movement information; the electronic device The electronic device is controlled according to the first eye movement type, and the electronic device is restricted from acquiring the user's gestures.

In a fifth aspect, the present application provides an electronic device, including: an eye-tracking device for acquiring user's eye-movement information; a gesture tracking device for acquiring user's gestures; at least one memory for storing programs; at least one The processor is used to execute the program stored in the memory. When the program stored in the memory is executed, the processor is used to: control the eye movement tracking device to obtain the eye movement information of the user; when the eye movement type of the user's eyes determined by the eye movement information is For the first type, the electronic device is controlled according to the first type; when the eye movement type of the user’s eyes determined by the eye movement information is the second type, the electronic device is controlled to instruct the user to perform gesture control; the gesture tracking device is controlled to obtain the user’s Gesture: controlling the electronic device according to the gesture acquired by the gesture tracking device.

According to the fifth aspect, or any implementation manner of the fifth aspect above, the processor controls the electronic device to instruct the user to perform gesture control, specifically including: the processor determines the user's gaze position according to the user's eye movement information; the processor controls the electronic device to The device prompts the user to perform gesture control within the first preset area where the gaze position is located.

According to the fifth aspect, or any implementation manner of the above fifth aspect, the processor controls the electronic device to instruct the user to perform gesture control, specifically including: the processor determines the target area according to the type of the first content displayed on the electronic device; The processor controls the electronic device to prompt the user to perform gesture control in a target area, wherein the target area is an area on the electronic device that does not contain the first content.

According to the fifth aspect, or any implementation of the fifth aspect above, the processor controls the electronic device to instruct the user to perform gesture control, specifically including: the processor obtains the user's gaze position according to the user's eye movement information; The location determines the gaze area of the user; when the gaze area is the second preset area, the processor controls the electronic device to prompt the user to perform gesture control.

According to the fifth aspect, or any implementation manner of the above fifth aspect, the processor controls the electronic device to instruct the user to perform gesture control, including one or more of the following: the processor controls the electronic device to display a cursor, and the processor controls the electronic device to zoom in At least part of the content is displayed, or the processor controls a sound component matched with the electronic device to broadcast a first voice, and the first voice is used to prompt the user to perform gesture control.

According to the fifth aspect, or any implementation of the above fifth aspect, the processor controlling the electronic device to instruct the user to perform gesture control includes the processor controlling the electronic device to display a cursor, and the cursor is displayed in the first area of the display area of the electronic device Inside; the processor is also used for: when the eye movement type of the user's eyes determined by the eye movement information continues to be the second type, and the user's eyes move and the user's gaze area is determined by the first area in the display area of the electronic device When switching to the second area, the electronic device is controlled to move the cursor from the first area to the second area, wherein the gaze area is determined by the gaze position of the user's eyes.

According to the fifth aspect, or any implementation of the above fifth aspect, the processor controlling the electronic device to instruct the user to perform gesture control includes the processor controlling to enlarge at least part of the content displayed by the electronic device, and the enlarged content is the display of the electronic device The content displayed in the first area in the area; the processor is also used for: when the eye movement type of the user's eyes determined by the eye movement information continues to be the second type, and the user's eyes move and the user's gaze area is determined by the electronic When the first area of the device is switched to the second area, the electronic device is controlled to restore the enlarged content in the first area to an initial state, and to enlarge the content in the second area.

According to the fifth aspect, or any implementation manner of the above fifth aspect, after the processor controls the electronic device to instruct the user to perform gesture control, the processor is further configured to: when the eye movement type of the user's eyes determined by the eye movement information is determined by the first When the second type is switched to the first type, the control electronic device stops instructing the user to perform gesture control, and/or controls the electronic device according to the first type, and restricts the gesture tracking device from acquiring the user's gestures.

According to the fifth aspect, or any implementation of the fifth aspect above, after the eye movement type of the user's eyes determined by the eye movement information is the second type, the processor is also used to: obtain the user's gaze position; The position determines the size of the user's gaze area; according to the size of the gaze area, the size of the operation area when the user performs gesture control is determined.

According to the fifth aspect, or any implementation manner of the above fifth aspect, the processor controls the electronic device according to the first type, specifically including: the processor determines the gaze position of the user in real time according to the first type, and/or, The processor controls the electronic device to switch the content displayed on the electronic device according to the first type.

In a sixth aspect, the present application provides an electronic device, including: an eye-tracking device for acquiring user's eye-movement information; a gesture tracking device for acquiring user's gestures; at least one memory for storing programs; at least one The processor is used to execute the program stored in the memory. When the program stored in the memory is executed, the processor is used to: control the user's gesture acquired by the gesture tracking device; control the electronic device according to the gesture acquired by the gesture tracking device; When the eye movement type of the user's eyes determined by the eye movement information is the first type, control the electronic device according to the first type, and limit the control of the electronic device according to the user's gesture; when the eye movement type of the user's eyes determined by the eye movement information is In the second type, the electronic device is controlled according to the gesture acquired by the gesture tracking device.

According to the sixth aspect, or any implementation of the sixth aspect above, after the processor controls the electronic device according to the first type and restricts the control of the electronic device according to the user's gesture, it is further used to: when the eye movement information When the determined eye movement type of the user's eyes is switched from the first type to the second type, control the electronic device to instruct the user to perform gesture control; control the user's gesture acquired by the gesture tracking device; control the electronic device according to the gesture acquired by the gesture tracking device .

According to the sixth aspect, or any implementation of the sixth aspect above, the processor restricts controlling the electronic device according to the user's gestures, specifically including: the processor controls the gesture tracking device not to acquire the user's gestures, or the processor controls the gestures The tracking device continues to acquire the user's gestures but does not process the gestures.

In a seventh aspect, the present application provides an electronic device, including: an eye-tracking device for acquiring user's eye-movement information; a gesture tracking device for acquiring user's gestures; at least one memory for storing programs; at least one The processor is used to execute the program stored in the memory. When the program stored in the memory is executed, the processor is used to: control the gesture tracking device to obtain the user's gesture; according to the user's operation on the first area on the electronic device obtained by the gesture tracking device The first gesture controls the electronic device; controls the eye movement tracking device to obtain the user's eye movement information; when the eye movement type of the user's eyes determined by the eye movement information is the first type, controls the electronic device according to the first type, And limit the gesture tracking device to obtain the user's gesture; continue to control the eye tracking device to obtain the user's eye movement information; when the eye movement type of the user's eyes determined by the eye movement information is switched from the first type to the second type, and the user's gaze When the location is switched from the first area on the electronic device to the second area, the electronic device is controlled to instruct the user to perform gesture control; the gesture tracking device is controlled to obtain the user's gesture; Gestures control electronic devices.

In an eighth aspect, the present application provides an electronic device, including: an eye-tracking device for acquiring user's eye-movement information; a gesture tracking device for acquiring user's gestures; at least one memory for storing programs; at least one The processor is used to execute the program stored in the memory. When the program stored in the memory is executed, the processor is used to: control the eye movement tracking device to obtain the user's first eye movement information; determine the first eye movement type according to the first eye movement information ; Control the electronic device according to the first eye movement type; control the eye movement tracking device to obtain the user's second eye movement information; determine the second eye movement type according to the second eye movement information; control the gesture tracking device to obtain the user's gesture; The gestures acquired by the gesture tracking device control the electronic equipment.

According to the eighth aspect, or any implementation of the above eighth aspect, the processor is further configured to: control the eye-tracking device to obtain the user's third eye-movement information; determine the first eye-movement type according to the third eye-movement information; The electronic device is controlled according to the type of the first eye movement, and the gesture tracking device is restricted from acquiring the user's gestures.

In the ninth aspect, the present application provides a computer-readable storage medium. The computer-readable storage medium stores a computer program. When the computer program runs on the electronic device, the electronic device executes any one of the first to fourth aspects. Aspect or the method provided in any implementation manner of the first aspect to the fourth aspect.

In a tenth aspect, the present application provides a computer program product. When the computer program product runs on an electronic device, the electronic device executes any one of the first to fourth aspects or any of the first to fourth aspects. A method provided in an implementation.

It can be understood that, for the beneficial effects of the above fifth aspect to the tenth aspect, reference may be made to the relevant description in the above first aspect, which will not be repeated here.

Description of drawings

FIG. 1 is a schematic diagram of an application scenario provided by an embodiment of the present application;

FIG. 2 is a schematic structural diagram of an electronic device provided by an embodiment of the present application;

Fig. 3 is a schematic diagram of eye movement tracking provided by an eye tracking device according to an embodiment of the present application;

Fig. 4 is a schematic diagram of the relative position between the pupil center of the user's eye and the light spot on the cornea provided by an embodiment of the present application;

Fig. 5 is a schematic diagram of the coordinates of the gaze position on the display screen of an electronic device provided by an embodiment of the present application;

Fig. 6 is a schematic layout diagram of an infrared emitter on an electronic device provided by an embodiment of the present application;

Fig. 7 is a schematic diagram of a gesture tracking device tracking gestures provided by an embodiment of the present application;

Fig. 8 is a schematic flowchart of a method for interacting with an electronic device provided by an embodiment of the present application;

Fig. 9 is a schematic diagram of instructing a user to perform gesture control provided in an embodiment of the present application;

Fig. 10 is another schematic diagram of instructing a user to perform gesture control provided in an embodiment of the present application;

Fig. 11 is a schematic diagram of the change of the cursor displayed on the electronic device when the user's gaze area changes according to an embodiment of the present application;

Fig. 12 is a schematic diagram of changes in content enlarged on an electronic device when a user's gaze area changes according to an embodiment of the present application;

Fig. 13 is a corresponding schematic diagram of an operation area and a display screen on an electronic device provided by an embodiment of the present application;

Fig. 14 is a corresponding schematic diagram of an operation area and a gaze area on an electronic device according to an embodiment of the present application;

Fig. 15 is a schematic diagram of an eye's viewing angle provided by an embodiment of the present application;

Fig. 16 is a schematic diagram of displaying a cursor in a gaze area on an electronic device according to an embodiment of the present application;

Fig. 17 is a schematic diagram of the steps of a control process provided by an embodiment of the present application;

Fig. 18 is a schematic diagram of a user's hand movement process provided by an embodiment of the present application;

Fig. 19 is a schematic flowchart of a method for interacting with an electronic device provided by an embodiment of the present application;

Fig. 20 is a schematic flowchart of another method for interacting with an electronic device provided by an embodiment of the present application;

Fig. 21 is a schematic flowchart of another method for interacting with an electronic device provided by an embodiment of the present application;

Fig. 22 is a schematic flowchart of another method for interacting with an electronic device provided by an embodiment of the present application;

Fig. 23 is a schematic diagram of a control process provided by an embodiment of the present application;

Fig. 24 is a schematic diagram of another control process provided by an embodiment of the present application;

Fig. 25 is a schematic diagram of another control process provided by an embodiment of the present application;

Fig. 26 is a schematic diagram of another control process provided by an embodiment of the present application;

Fig. 27 is a schematic diagram of another control process provided by an embodiment of the present application.

Detailed ways

The term "and/or" in this article is an association relationship describing associated objects, which means that there can be three relationships, for example, A and/or B can mean: A exists alone, A and B exist simultaneously, and B exists alone These three situations. The symbol "/" in this document indicates that the associated object is an or relationship, for example, A/B indicates A or B.

The terms "first" and "second" and the like in the specification and claims herein are used to distinguish different objects, rather than to describe a specific order of objects. For example, the first response message and the second response message are used to distinguish different response messages, rather than describing a specific order of the response messages.

In the embodiments of the present application, words such as "exemplary" or "for example" are used as examples, illustrations or illustrations. Any embodiment or design scheme described as "exemplary" or "for example" in the embodiments of the present application shall not be interpreted as being more preferred or more advantageous than other embodiments or design schemes. Rather, the use of words such as "exemplary" or "such as" is intended to present related concepts in a concrete manner.

In the description of the embodiments of the present application, unless otherwise specified, "multiple" means two or more, for example, multiple processing units refer to two or more processing units, etc.; multiple A component refers to two or more components or the like.

Exemplarily, Fig. 1 shows an application scenario in some embodiments of the present application. As shown in Figure 1, user A is using an electronic device 100, wherein the electronic device 100 may be, but not limited to, a smart TV. A TV or other electronic devices with a large screen for interaction, such as a user interface in a smart phone can be transmitted wirelessly and presented on the smart TV, and user operations on the smart TV can also affect the smart phone. The electronic device 100 may be provided with an eye tracking device 110 and/or a gesture tracking device 120 . The eye tracking device 110 can be used to detect the movement of the user A's eyes 21 , and the gesture tracking device 120 can be used to detect the gesture and/or hand movement of the user A's hand 22 .

Generally, user A can use eye movement manipulation to control the electronic device 100 . At this time, an eye tracking device 110 may be provided on the electronic device 100 . Wherein, during the process of using the electronic device 100 by the user A, the user A may intentionally perform some control-related actions through his eyes 21 . Afterwards, after the eye-tracking device 110 detects the motion of the eye 21 , the electronic device 100 may generate a control command according to the motion of the eye 21 detected by the eye-tracking device 110 , and output the control command. It can be seen that during the manipulation of the electronic device 100 by the user A, the eyes 21 of the user A need to make frequent deliberate movements, which easily causes eye fatigue of the user A.

In order to avoid eye fatigue caused by eye movement manipulation, eye movement manipulation may be replaced by gesture manipulation. At this time, the eye tracking device 110 on the electronic device 100 can be replaced by the gesture tracking device 120 . At this time, during the process of using the electronic device 100 by the user A, the user A may intentionally make some control-related gestures through his hand 22 . Afterwards, after the gesture tracking device 120 detects the gesture made by the hand 22, the electronic device 100 can generate a control command according to the gesture made by the hand 22 detected by the gesture tracking device 120, and output the control command. However, when the control area on the electronic device 100 is large, the distance that user A's hand 22 needs to move will also increase, which will easily lead to a decrease in control efficiency and cause fatigue.

Further, in order to avoid the problems of low efficiency and easy fatigue caused by separate eye movement control and gesture control, and realize efficient and convenient control of electronic devices, the present application also provides a solution. Among them, in this solution, the electronic device 100 can be provided with an eye tracking device 110 and a gesture tracking device 120. At this time, when user A uses the electronic device 100, the eye tracking device 110 on the electronic device 100 The eyes 21 of user A can be detected, and when it is detected that user A intends to control the electronic device 100, the electronic device 100 can instruct user A to perform gesture control; The gesture of A's hand 22 generates a control command, and outputs the control command. In this way, eye movement control and gesture control are combined in an orderly manner, so that there can be no sense interaction between eye movement control and gesture control, and efficient, convenient and high-precision control of electronic devices is realized.

It can be understood that the electronic device 100 shown in FIG. 1 can also be replaced with other electronic devices, and the replaced solution is still within the protection scope of the present application. Exemplarily, the electronic device 100 can be a mobile phone, a tablet computer, a desktop computer, a laptop computer, a handheld computer, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, and a cell phone, a personal Digital assistant (personal digital assistant, PDA), augmented reality (augmented reality, AR) equipment, virtual reality (virtual reality, VR) equipment, artificial intelligence (artificial intelligence, AI) equipment, wearable equipment, vehicle equipment, smart home device and/or smart city device, the embodiment of the present application does not specifically limit the specific type of the electronic device 100 .

Exemplarily, FIG. 2 shows a schematic structural diagram of the electronic device 100 . As shown in FIG. 2 , the electronic device 100 may include: an eye tracking device 110 , a gesture tracking device 120 , a processor 130 and a memory 140 .

Wherein, the eye movement tracking device 110 may be used to detect eye movement information, eye movement type and/or gaze position of the user's eyes. Exemplarily, the eye movement information may include eye movement velocity and/or eye movement acceleration and the like. Exemplarily, the type of eye movement may include bouncing and non-bouncing, wherein non-bouncing refers to an eye movement type other than bouncing, such as gaze or smooth tracking. The eye-tracking device 110 may, but is not limited to, detect eye movement types through pupil center corneal reflection (pupil center corneal reflection, PCCR), appearance-based and other technologies. Exemplarily, taking PCCR as an example, as shown in FIG. 3 , the eye tracking device 110 may include: an infrared emitter 111 and a camera 112 . When the eye tracking device 110 is in operation, the infrared emitter 111 can emit infrared light, which can be reflected on the cornea of user A's eye 21 and project light spots on the cornea. The camera 112 can collect the face image of user A, and determine the relative position between the pupil center of the user's eye 21 and the light spot on the cornea through a preset image algorithm, and determine the gaze of user A's eye 21 based on the relative position Point and eye movement types. Exemplarily, the eye-tracking device 110 can be started together with the electronic device 100, or can be started after obtaining the start instruction. Exemplarily, when an image capture device (such as a camera) is configured on the electronic device 100, and the electronic device 100 determines that the image capture device has captured a user's face image, the electronic device 100 may send an image to the eye tracking device 110 Start command. In addition, the electronic device 100 may also send an activation instruction to the eye tracking device 110 after obtaining the instruction from the user to start the eye tracking device 110 . In addition, when the electronic device 100 obtains an instruction from the user to turn on the electronic device 100 , the electronic device 100 may also send an activation instruction to the eye tracking device 110 .

In one example, the mapping relationship between the relative position of the pupil center of the user's eye and the light spot on the cornea and the gaze point of the user's gaze on the electronic device 100 can be preset, so that when the pupil center of the user's eye and the cornea After determining the relative positions between the light spots, the gaze position of the user on the electronic device 100 can be determined. Exemplary, as shown in (A) of Figure 4, if the pupil center 41 of the user's eye 21 and the infrared light emitted by the infrared emitter 111 coincide with the light spot 42 transmitted on the cornea of the user's eye 21, the user's gaze direction is Staring at the central area of the electronic device 100, when it is determined that the pupil center of the user's eye a coincides with the light spot on the cornea, it can be determined that the user is looking at the central area of the electronic device 100, wherein, as shown in Figure 5, the center The coordinates of the region can be (0,0). As shown in (B) of Figure 4, if the pupil center 41 of the user's eye 21 is located at the right left of the light spot 42 transmitted by the infrared light emitted by the infrared emitter 111 on the cornea of the user's eye 21, and the distance between the two is L When ₁ , the user's gaze direction is to gaze at the central area of the right edge of the electronic device 100, then when it is determined that the pupil center 41 of the user's eye 21 is located directly to the left of the transmitted light spot 42 on the cornea, and the distance between the two is L ₁ , it can be determined that the user is looking at the central area of the right edge of the electronic device 100, where, as shown in FIG. 5 , the coordinates of this area can be (X ₁ ,0). As shown in (C) of Figure 4, if the pupil center 41 of the user's eye 21 is located at the right side of the light spot 42 transmitted by the infrared light emitted by the infrared emitter 111 on the cornea of the user's eye 21, and the distance between the two is L ₁ , the user's gaze direction is to gaze at the central area of the left edge of the electronic device 100, then when it is determined that the pupil center 41 of the user's eye 21 is located directly to the right of the transmitted light spot 42 on the cornea, and the distance between the two is L ₁ , it can be determined that the user is looking at the central area of the left edge of the electronic device 100, where, as shown in FIG. 5 , the coordinates of this area can be (-X ₁ ,0).

In an embodiment, when determining the type of eye movement, it may be determined based on, but not limited to, eye movement velocity, eye movement acceleration, or spatial divergence. Exemplarily, the determination of the eye movement type based on the eye movement speed is used as an example for illustration. When the eye movement speed of the user's eyes exceeds a certain threshold V ₁ , that is, the eye movement speed V ∈ (V ₁ , +∞), it indicates that the user's eye movement If the eye movement speed is fast, it can be determined that the eye movement type is jumping; when the eye movement speed of the user's eyes is between V ₂ and V ₁ , that is, the eye movement speed V ∈ [V ₂ , V ₁ ], it indicates that the user's eye movement speed is between V 2 and V 1 . The eye movement speed is relatively slow. At this time, it can be determined that the eye movement type is gentle tracking, where V ₂ is less than V ₁ ; when the eye movement speed of the user's eyes is less than V ₂ , that is, the eye movement speed V∈[0, V ₂ ), it indicates The user's eye movement speed is very slow, and the eye movement type can be determined to be fixation at this time.

In other embodiments, the number of infrared emitters 111 may be multiple. A plurality of infrared emitters 111 may be arranged on the electronic device 100 at intervals. For example, as shown in FIG. 6 , when the number of infrared emitters 111 is four, the four infrared emitters 111 may be respectively arranged at four corners of the electronic device 100 . In addition, the number of cameras 112 may also be multiple, which may be determined according to specific circumstances, and is not limited here. In an example, at least part of the functions in the eye tracking device 110 (such as data processing functions, etc.) may be implemented by the processor 110 . In an example, the eye-tracking device 110 may be integrated on the electronic device 100 or arranged separately, which is not limited here.

The gesture tracking device 120 can be used to detect gestures made by the user. The gesture tracking device 120 may, but not limited to, detect the gestures made by the user through methods such as camera tracking based on computer vision, comparison based on transmitted signals and reflected signals, and the like. Exemplarily, taking the comparison method based on transmitted signals and reflected signals as an example, as shown in FIG. 7 , the gesture tracking device 120 may include a signal transmitter 121 , a signal receiver 122 and a signal processing unit 123 . During the working process of the gesture tracking device 120, the signal transmitter 121 can send signals (such as signals such as millimeter wave, infrared light, ultrasound, wireless fidelity (Wi-Fi)); then, the signal receiver 122 can receive The signal reflected by the hand 22 of user A; finally, the signal processing unit 123 compares the original signal sent by the signal transmitter 121 with the reflected signal received by the signal receiver 122, and utilizes principles such as the Doppler effect, phase shift, and time difference The movement of the hand 22 of the user A is tracked, and then the gesture made by the user A is determined. In one example, when the gesture tracking device 120 uses a camera tracking method based on computer vision to detect the gestures made by the user, the gesture tracking device 120 may also include a camera, which can collect images of the user's hand and A set image processing algorithm detects gestures made by the user. Wherein, the camera may be the same camera as the camera in the eye tracking device 110, or may be a different camera. In one example, at least some components in the gesture tracking device 120 (such as the signal processing unit 123 ) can be integrated in the processor 130 . In one example, the gesture tracking device 120 may be integrated on the electronic device 100 or arranged separately, or a part may be integrated on the electronic device 100 while another part is arranged separately, which is not limited here. In one example, at least part of the functions (such as data processing functions, etc.) of the gesture tracking device 120 may be implemented by the processor 110 . Exemplarily, the gesture tracking device 120 can be started together with the electronic device 100, or can be started after obtaining the start command, for example, when the electronic device 100 determines that the eye movement type of the user's eyes is not When jumping, the electronic device 100 may send an activation instruction to the gesture tracking device 120 . In addition, the electronic device 100 may also send an activation instruction to the eye tracking device 120 after obtaining the instruction from the user to start the eye tracking device 120 . In addition, when the electronic device 100 obtains an instruction from the user to turn on the electronic device 100 , the electronic device 100 may also send an activation instruction to the eye tracking device 120 .

In one embodiment, the gesture tracking device 120 may include peripheral components such as an electromyogram (electromyogram, EMG) bracelet or an infrared remote control pen. In addition, when the user can use the peripheral components in the gesture tracking device 120, the signal processing unit in the gesture tracking device 120 can obtain the signals emitted by the peripheral components, and detect the user's gestures and gestures according to the signals emitted by the peripheral components. sports.

In one embodiment, during the working process of the gesture tracking device 120 , the eye tracking device 110 may also be in a working state, so that the eye tracking device 110 continues to acquire the eye movement type of the user's eyes.

In one embodiment, when the eye tracking device 110 is working, the gesture tracking device 120 can stop working to save power consumption, or the gesture tracking device 120 can also continue to work so that the gesture tracking device 120 can continue to acquire the user's gestures .

The processor 130 is a calculation core and a control core of the electronic device 100 . Processor 130 may include one or more processing units. For example, the processor 130 may include an application processor (application processor, AP), a modem (modem), a graphics processing unit (graphics processing unit, GPU), an image signal processor (image signal processor, ISP), a controller, a video encoder One or more of a decoder, a digital signal processor (digital signal processor, DSP), a baseband processor, and/or a neural network processor (neural-network processing unit, NPU), etc. Wherein, different processing units may be independent devices, or may be integrated in one or more processors. Exemplarily, the processor 130 may issue a control command to the eye tracking device 110 and/or the gesture tracking device 120, for example: controlling the eye tracking device 110 to be turned on or off, so as to control the eye tracking device 110 to obtain the user's eye movement Information or stop acquiring the user's eye movement information, etc., control the gesture tracking device 120 to start or shut down, so as to control the gesture tracking device 120 to acquire the user's gestures or stop acquiring the user's gestures. Exemplarily, the processor 130 may determine the user's eye movement type according to the user's eye movement information acquired by the eye movement tracking device 110. For details, refer to the above-mentioned description of determining the eye movement type based on the eye movement information, which will not be repeated here. A repeat.

The memory 140 can store a program, and the program can be executed by the processor 130, so that the processor 130 can at least execute the method provided in the embodiment of the present application. The memory 140 may also store data. The processor 130 can read data stored in the memory 140 . The memory 140 and the processor 130 may be provided separately. In addition, the memory 140 may also be integrated in the processor 130 .

In other embodiments, the electronic device 100 further includes a display screen (not shown in the figure). The display screen can be used to display images, videos, etc. The display screen may include a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active matrix organic light emitting diode or an active matrix organic light emitting diode (active-matrix organic light emitting diode, AMOLED), flexible light-emitting diode (flex light-emitting diode, FLED), Miniled, MicroLed, Micro-oLed, quantum dot light emitting diodes (quantum dot light emitting diodes, QLED), etc. In some embodiments, the electronic device 100 may include 1 or N display screens, where N is a positive integer greater than 1. Exemplarily, the electronic device may display information on the display screen indicating that the user performs gesture control.

It can be understood that the structure shown in FIG. 2 of the present application does not constitute a specific limitation on the electronic device 100 . In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown in the figure, or combine certain components, or separate certain components, or arrange different components. The illustrated components can be realized in hardware, software or a combination of software and hardware.

In some embodiments, the electronic device 100, such as a smart TV, may display a user interface as shown in (A) of FIG. Eye movement type, for example: detection of eye movement type through pupil center corneal reflection technology, appearance detection and other technologies (see the above description for details, and will not be repeated here). When the type of eye movement acquired by the eye-tracking device 110 is not jitter (that is, the type of eye movement is non-jitter), the electronic device 100 may instruct the user to perform gesture control. For example, as shown in (B) of FIG. 9 , the electronic device 100 may display a cursor 92 to direct the user to gesture control. In addition, when the type of eye movement acquired by the eye-tracking device 110 is not bouncing, the electronic device 100 can activate the gesture tracking device 120 on it to acquire the user's operation gesture, and then respond to the user's operation.

Based on the content described above and in conjunction with FIG. 8 , the technical solution provided by the present application will be described in detail below. Exemplarily, FIG. 8 shows a schematic flowchart of a method for interacting with an electronic device provided in an embodiment of the present application. The electronic device involved in FIG. 8 may be the electronic device 100 described above in FIG. 2 , which has the eye tracking device 110 and the manual tracking device 120 described above. As shown in Figure 8, the method for interacting with an electronic device may include the following steps:

S801. Obtain an eye movement type of the user's eyes. The eye movement type may include jitter or non-jitter, and non-jitter means that the eye movement type is not jitter.

Specifically, the eye movement type of the user's eyes may be acquired in real time or periodically by using the eye movement tracking device 110 described above. Wherein, the type of eye movement may include jumping or non-beating, and non-beating means that the type of eye movement is not jumping, such as gaze or smooth tracking. In addition, the processor 130 in the electronic device 100 can also determine the eye movement type of the user's eyes based on the eye movement information of the user's eyes obtained by the eye movement tracking device 110 in real time or periodically. Do limited.

S802. Determine that the eye movement type is non-jumping, and instruct the user to perform gesture control.

Specifically, after the eye movement type is acquired, the user can be instructed to perform gesture control according to the eye movement type, so that the user can learn that the electronic device can be controlled by gestures in the future. In some embodiments, when it is determined that the eye movement type is not jerk (ie, gaze or gentle tracking, etc.), the user may be instructed to perform gesture control. When it is determined that the eye movement type is jitter, the detection of the user's eye movement type may continue, and at this time, the display interface on the electronic device may not change. In addition, when the eye movement type at the previous moment is not jerk, but the eye movement type at the current moment is jerk, the user is no longer instructed to perform gesture control. That is to say, when the eye movement type of the user's eyes switches from non-jumping to jittering, the user is no longer instructed to perform gesture control. When the eye movement type of the user's eyes is switched from bouncing to non-bouncing, the user is instructed to perform gesture control. Here, non-jump means not jitter.

Exemplarily, as shown in (A) of FIG. 9 , the electronic device 100 has a display screen, and the display screen is in a bright state, and an icon of an application or other programs that the user is watching may be displayed on the display screen of the electronic device 100 . Content (such as text, pictures, etc.); at this time, taking the application icon displayed on the display screen as an example, when instructing the user to perform gesture control, as shown in (B) of FIG. 9 , the display on the electronic device 100 A cursor 92 may be displayed on the screen, so as to indicate the user to perform gesture control through the cursor 92 . Exemplarily, as shown in (A) of FIG. 10 , the electronic device 100 has a display screen, and the display screen is in a bright state, and the display screen of the electronic device 100 may display application icons or other programs that the user is viewing. Content (such as text, pictures, etc.), when instructing the user to perform gesture control, as shown in (B) of FIG. to instruct the user to perform gesture controls.

In some embodiments, referring to (B) of FIG. 9 , the position of the cursor 902 may be the starting position of the gesture operation. Continuing to refer to (B) of FIG. 10 , the position of the enlarged icon 901 may be the starting position of the gesture operation. Exemplarily, the starting position of the gesture operation can be understood as the initial moving position of the cursor when the user performs the gesture operation.

It can be understood that, in addition to using the instruction methods shown in FIG. 9 and FIG. 10 , voice instructions and other methods can also be used to instruct the user to perform gesture control, which may be determined according to actual conditions, and is not limited here.

S803. Obtain an operation gesture of the user.

Specifically, the gesture tracking device 120 described above may be used to obtain the user's operation gestures, see the above description for details, and will not be repeated here. Exemplarily, an operation gesture may also be referred to as a gesture.

S804. Control the electronic device according to the operation gesture.

Specifically, after the user's operation gesture is acquired, the electronic device can be controlled according to the operation gesture. Exemplarily, after the operation gesture is obtained, the mapping relationship between the preset gesture and the control instruction can be queried according to the operation gesture, so as to determine the control instruction corresponding to the operation gesture, and then based on the control instruction, the electronic device can be controlled. control.

Therefore, through the orderly combination of eye movement control and gesture control, there can be no sense interaction between eye movement control and gesture control, so that the user can control the electronic device efficiently, conveniently and with high precision.

In some embodiments, before instructing the user to perform gesture control in the above S802, the gaze position of the user's eyes can be determined in real time according to the eye movement type during jumping, so as to determine the user's potential control area, and then the user can be accurately instructed For gesture control. Wherein, the way of determining the gaze position of the user's eyes can refer to the relevant introduction about the eye tracking device 110 in FIG. 2 above, and will not be repeated here. Exemplarily, the gaze position of the user's eyes may represent the user's interest area. Exemplarily, "determining the gaze position of the user's eyes" can be understood as "controlling the electronic device according to the type of eye movement during jumping".

Exemplarily, when precisely instructing the user to perform gesture control, the user may be instructed to perform gesture control within the determined preset area where the gaze position of the user's eyes is located. For example: continue to refer to (A) of FIG. 9 , when it is determined that the user's gaze position is the position where the icon 91 is, as shown in (B) of FIG. 9 , the cursor 92 can be displayed at the position where the icon 91 is. Similarly, continuing to refer to (A) of FIG. 10 , when it is determined that the user's gaze position is where the icon 91 is located, as shown in (B) of FIG. 10 , the icon 91 can be enlarged. Since the user's gaze position can indicate the user's area of interest, this can indicate that the user can perform gesture control at the same time, so that the user can directly operate the area of interest, avoiding the need for the user to reselect the desired area. Controlled content improves user experience.

In some embodiments, when the eye movement type of the user's eyes is twitching, the content displayed on the electronic device may be switched according to the current eye movement type. Exemplarily, the user interface displayed on the electronic device can be controlled to be switched. Wherein, the switching direction may be determined by the changing direction of the gaze position of the user's eyes. For example, when the user is browsing a file, the file has 10 pages in total, and the electronic device can only display 1 page at a time, and the user is browsing page 5, when it is determined that the eye movement type of the user's eyes is twitching, and the user's eyes If the gaze position of the electronic device moves from the top to the bottom, then when the gaze position of the user's eyes is at the bottom of the display area on the electronic device, the electronic device can switch the content of the file it displays from page 5 to page 5. Page 6; when it is determined that the eye movement type of the user's eyes is jitter, and the gaze position of the user's eyes is moving from the bottom to the top of the electronic device, then when the gaze position of the user's eyes is at the top of the display area on the electronic device, The electronic device can switch the content of the displayed file from page 5 to page 4. Exemplarily, "when the eye movement type of the user's eyes is bouncing, control and switch the content displayed on the electronic device according to the eye movement type at this time" can be understood as "controlling the electronic device according to the eye movement type of the bouncing control".

In some embodiments, when the electronic device is controlled according to the type of eye movement during jumping, the electronic device may be restricted from acquiring the user's gestures, so as to avoid interference from the user's gestures. Exemplarily, restricting the electronic device from acquiring the user's gesture may include: the electronic device does not acquire the user's gesture, or, although the electronic device acquires the user's gesture, it does not process the gesture. As a possible implementation manner, the gesture tracking device 120 can be controlled to stop working, or the gesture tracking device 120 can be continuously controlled to work, but the acquired gestures are not processed.

In some embodiments, considering that the user generally needs to focus on the item when operating an item, in order to facilitate the user's manipulation, when instructing the user to perform gesture control, the user's eye movement type can also be skipped. , when the gaze area of the user on the display area of the electronic device 100 changes, the content used to instruct the user to perform gesture control (such as Cursor, etc.) until the user's eye movement type is gazing, so as to inform the user of his current gaze position, so as to facilitate the user to select an item. Exemplarily, taking the content used to instruct the user to perform gesture control as a cursor as an example, as shown in (A) of FIG. When the area 1101 on the display screen, the cursor 92 can be displayed in the area 1101 on the display screen of the electronic device 100 to instruct the user to perform gesture control. Then, when the user's eye movement type is gentle tracking, and the gaze area moves to When there is an area 1102 on the display screen of the electronic device 100 , as shown in (B) of FIG. 11 , the cursor 92 can be controlled to move from the area 1101 to the area 1102 . Exemplarily, taking the zoom-in content as an example to instruct the user to perform gesture control, as shown in (A) of FIG. When the area 1201 on the display screen is displayed, the icon 1202 in the enlarged area 1201 can be controlled to instruct the user to perform gesture control. Then, when the user's eye movement type is gentle tracking, and the gaze area moves to the display of the electronic device 100 When the area 1203 on the screen is displayed, as shown in (B) of FIG. 12 , the icon 1202 in the area 1201 can be controlled to return to the initial state (that is, the state not enlarged), and the icon 1204 in the enlarged area 1203 can be controlled.

In some embodiments, when the user's hand needs to move a large distance during the gesture control process of the user, it is easy to cause fatigue of the user's hand, resulting in poor user experience. In addition, as shown in FIG. 13 , when the hand 22 of the user A can move in the area 50 to complete the control of the electronic device 100, although it can solve the situation that the hand of the user A is easily fatigued due to the large movement of the hand, However, since the display area 51 on the electronic device 100 is obviously larger than the area 50, this makes the moving distance of the hand 22 of the user A in the area 50 often move a relatively large distance when it is mapped to the display area 51 on the electronic device 100, thereby causing control The decreased precision makes it difficult for the user to precisely control the electronic device 100 . In order to avoid the situation that the user's hand moves a large distance, and to improve the control accuracy, after the user's gaze position is determined, the user's gaze area can be determined based on the user's gaze position, and the size of the gaze area can be set in advance Correspondence with the size of the operation area of the user's hand, for example, the ratio between the size of the gaze area and the size of the operation area can be 1:1, or 1:0.8, or 1:1.2, etc. etc., so that the user's hand can operate in the operating area of the user's hand during operation (of course, the user's hand can also exceed the operating area during operation. At this time, the eye movement type of the user's eyes does not When it is not jumping, the cursor and other content used to instruct the user to perform gesture control can follow the user's hand to move outside the gaze area), and then complete the control of the content in the gaze area, improving the comfort of operation. For example, as shown in Figure 14, the user's gaze position is position x, and the gaze area determined based on position x is area 53 at this time, wherein, area 53 is mapped to the operation area at the position where user A is located as area 52, and then , user A can control his hand 22 to make a control gesture in this area 52, that is, he can control the content in the area 53, so that the moving distance of the user's hand can be controlled in a more comfortable area for the user , which improves the user's operating experience. In addition, in FIG. 14 , the size of the area 53 and the area 52 can be the same, so that the moving distance of the hand 22 of the user A in the area 52 can be mapped to the area 53 at a 1:1 ratio, and thus accurate Controlling the electronic device 100 improves the control precision. In one example, the size of the gaze area and the operation area is mainly to determine the scale of the operation. For example, when very high-precision manipulation is required, the ratio between the size of the gaze area and the operation area can be 2:1, etc.; When it is necessary to achieve lower-precision manipulation, the ratio between the size of the gaze area and the size of the operation area may be 1:2 or the like.

Exemplarily, the gaze area may be an area extending outward for a preset distance centered on the gaze position of the user. In addition, the gaze area may also be calculated in real time or periodically based on items such as the gaze position of the user, the distance between the user and the electronic device, or the angle at which the user watches the electronic device. Exemplarily, the distance between the user and the electronic device can be used to determine the target distance extending outward centered on the user's gaze position, and then the target distance can be obtained by extending the target distance outward centered on the user's gaze position. area.

For example, as shown in Figure 15, since the viewing angle A of the most sensitive area of the human eye is 2°, after obtaining the distance D between the user and the electronic device, the following "Formula 1" can be used After the target distance S is calculated, the target distance S can be extended outward with the user's gaze position as the center, so as to obtain the gaze area. Exemplarily, the gaze area may be a regular shape such as a circle or a rectangle, or may be an irregular shape.

Wherein, A=2; D is the distance between the user and the electronic device, D can be detected by the distance sensor on the electronic device; S is the target distance.

Further, after the gaze area is determined, information instructing the user to perform gesture control may be placed in the gaze area in S802 above. Exemplarily, as shown in FIG. 16 , when the determined gaze area is the area 54 and the cursor 92 is used to instruct the user to perform gesture control, the cursor 92 may be placed in the gaze area 54 . Of course, when using other indication methods to instruct the user to perform gesture control, the indication may also be performed in the determined gaze area, which will not be repeated here.

In some embodiments, when the cursor is used to instruct the user to perform gesture control, in order to prevent the cursor from blocking the content the user is watching, when the cursor is displayed, the cursor can be displayed at a position within the gaze area that does not block the user's line of sight. Exemplarily, the display position of the cursor in the gaze area may be determined according to the type of application currently being used by the user. For example, if the application that the user is currently using is a reading application, the cursor can be displayed at a position where no text is displayed in the gaze area when displaying the cursor; if the application that the user is currently using is a video application, at this time When displaying the cursor, the cursor may be displayed at the lower left corner of the gaze area, and so on.

In some embodiments, since the eye movement type of the user's eyes is not bouncing in most cases when watching a video, it is easy to instruct the user to perform gesture control in such a scene (that is, a movie viewing scene). Affect the user to watch the video. Therefore, in this scenario, in order to avoid misinstructing the user to perform gesture control, a preset area can be set on the electronic device. Gaze type directs the user to gesture control. Exemplarily, it may be determined whether the user's gaze area is a preset area based on the user's gaze position. For example, the user's gaze position may be compared with a preset area to determine whether the user's gaze position is located in the preset area. In addition, when performing this part of the operation, it is also possible to judge the scene in which the user uses the electronic device first, so as to determine whether the current use scene is a scene of watching a video. Wherein, the electronic device may determine the scene where the user uses the electronic device based on the type of the application (application, APP) being used by the user and the content displayed on the APP. For example, when the APP being used by the user is a video application and a video is being played on the APP, it can be determined that the scene where the user only has electronic devices is a scene of watching a video.

The above is the introduction of the method for interacting with the electronic device provided by the present application, and for the convenience of understanding, an example is given below for illustration.

Exemplarily, FIG. 17 shows a process of controlling an electronic device. In FIG. 17 , the electronic device is equipped with the eye tracking device and the gesture tracking device described above, wherein the cursor is used to indicate when instructing the user to perform gesture control. As shown in Figure 17, the process may include the following steps:

S1301. Perform eye movement tracking.

Specifically, the eye tracking device 110 described above may be used to perform eye tracking, see the above description for details, and details will not be repeated here.

S1302. Analyzing eye movement types.

Specifically, after the eye movement of the user's eyes is tracked, the eye movement type may be analyzed according to the tracked eye movement data. For example, according to the eye movement speed analysis of the user's eyes, etc. See the above description for details, and will not repeat them here.

S1303. Determine whether the eye movement type is jitter.

Specifically, after the eye movement type is determined, it can be determined whether the eye movement type is jitter. Wherein, when the eye movement type is jitter, it indicates that the current user has no intention of manipulating the electronic device. Therefore, at this time, S1304 is executed, that is, the execution returns to S1301. When the eye movement type is not jitter, it indicates that the current user has an intention to manipulate the electronic device, so at this time, the precise control stage can be entered, that is, S1305 is executed.

S1304. Determine the gaze position.

Specifically, when the user intends to manipulate the electronic device, the gaze position of the user may be determined, so as to subsequently determine the operation area. For the manner of determining the gaze position, refer to the above description for details, and details will not be repeated here.

After the fixation position is determined, subsequent procedures can be performed. Wherein, in the subsequent control stage, S1305 and S1310 may be executed simultaneously.

S1305. Determine a gaze area according to the gaze position, and display a cursor at an appropriate position.

Specifically, after the gazing position is determined, a gazing area can be determined according to the gazing position, and then a cursor can be displayed at an appropriate position in the gazing area. Exemplarily, the gaze area may be the area 53 described in FIG. 14 , and the display cursor may be the display manner described in FIG. 16 . See the above description for details, and will not repeat them here.

S1306. Perform hand tracking.

Specifically, the user's hand may be tracked by the gesture tracking device 120 described above, see the above description for details, and will not be repeated here.

S1307. The cursor moves along with the hand according to the prescribed scale.

Specifically, after the user's hand movement is tracked, the cursor can be controlled to follow the user's hand movement according to a specified scale. Exemplarily, as shown in FIG. 18 , at time T1, user A's hand 22 is at position 521 in area 52, and at this time cursor 92 is at position 531 in area 53; at time T2, user A's hand The hand part 22 moves and moves to the position 522 in the area 52 . At this time, the cursor 92 can move with the hand part 22 according to a prescribed scale and move to the position 532 in the area 53 .

S1308. Determine whether there is a gesture operation.

Specifically, the gesture tracking device 120 described above may perform hand tracking on the user's hand, and then determine whether there is a gesture operation. Wherein, when there is gesture operation, execute S1310; otherwise, return to execute S1307.

S1309. Complete the corresponding operation.

Specifically, when it is determined that there is a gesture operation, according to the preset correspondence between gestures and control instructions, a control instruction corresponding to the currently determined gesture is generated, and the control instruction is output to complete the corresponding operation . For example, continue to refer to (B) of FIG. 9 , at this time, when the determined gesture is a confirmation gesture, a confirmation instruction is output, thereby completing the operation of confirming and selecting "setting".

S1310. Determine whether the eye movement type is jitter, and if the eye movement type is jitter, control the cursor to disappear and return to execute S1301; if the eye movement type is not jitter, continue hand tracking, that is, execute S1306.

Specifically, in the gesture control stage, it is possible to continue to judge the user's eye movement type, and when the eye movement type is jumping, the control cursor disappears and returns to S1301, and when the eye movement type is not jumping, continue hand tracking , that is, execute S1306.

It can be understood that the sequence numbers of the steps in the embodiments of the present application do not mean the order of execution, and the execution order of each process should be determined by its functions and internal logic, and should not constitute the implementation process of the embodiments of the present application. Any restrictions. In addition, in some possible implementation manners, the execution order of each execution step in the embodiment of the present application may be adjusted and/or selectively executed according to actual conditions, which is not limited here. In addition, the eye movement control and gesture control mentioned in the embodiments of this application can be combined as needed, for example: eye movement control first and then gesture control, or gesture control first and then eye movement control, or eye movement control and gesture control Simultaneously etc.

As a possible implementation manner, as shown in FIG. 19 , at S1901, the electronic device may obtain an eye movement type of the user's eyes. At S1902, when it is determined that the eye movement type is the first type (for example: jitter), the electronic device controls the electronic device according to the first type. At S1903, when it is determined that the eye movement type is the second type (for example: non-jumping), the electronic device instructs the user to perform gesture control. Afterwards, at S1904, the electronic device may acquire the user's gesture. Next, at S1905, the electronic device may control the electronic device according to the acquired gesture. It can be understood that, for the process of obtaining the eye movement type and the gesture of the electronic device, and the process of controlling the electronic device, etc., please refer to the relevant description above, and will not repeat them here.

As another possible implementation manner, as shown in FIG. 20 , at S2001, the electronic device may acquire a user's gesture first. Then, at S2002, the electronic device is controlled according to the acquired gesture. Next, in S2003, the electronic device may obtain the user's eye movement type, wherein the electronic device may also execute S2003 during the process of obtaining the user's gesture or during the process of controlling the electronic device. Next, at S2004, when it is determined that the user's eye movement type is the second type (for example: non-jumping), the electronic device may continue to obtain the user's gesture, and at S2005, control the electronic device according to the obtained gesture. Or, in S2006, when it is determined that the user's eye movement type is the first type (for example: jumping), the electronic device can control the electronic device according to the eye movement type at this time, and limit the electronic device to control the electronic device according to the user's gesture , for example: the electronic device may not obtain the user's gesture, or the electronic device may not process the user's gesture although it has obtained the user's gesture. In an example, after the electronic device controls the electronic device according to the first type of eye movement type, if the electronic device determines that the eye movement type of the user's eyes is switched from the first type (such as: jumping) to the second type (such as : non-jumping), the electronic device can instruct the user to perform gesture control; then, the electronic device can obtain the gesture of the user, and control the electronic device according to the obtained gesture. It can be understood that, for the process of obtaining the eye movement type and the gesture of the electronic device, and the process of controlling the electronic device, etc., please refer to the relevant description above, and will not repeat them here.

As yet another possible implementation manner, as shown in FIG. 21 , at S2101, the electronic device may first obtain eye movement information of the user. In S2102, the eye movement type determined by the electronic device according to the eye movement information is the first type (for example: jumping), and then, in S2103, the electronic device may control the electronic device according to the eye movement type at this time. Next, at S2104, the electronic device continues to obtain the user's eye movement information, and then, at S2105, the electronic device determines that the eye movement type according to the eye movement information is the second type (for example: non-jumping), and then, at S2106, the electronic device The user's gesture can be acquired, and the electronic device can be controlled according to the acquired gesture at S2107. Afterwards, the electronic device can continue to acquire the user's eye movement information. When the eye movement type determined by the electronic device from the eye movement information is the first type (for example: jumping), the electronic device can control the electronic device according to the eye movement type at this time. Take control and restrict access to user gestures. It can be understood that, for the process of obtaining the eye movement type and the gesture of the electronic device, and the process of controlling the electronic device, etc., please refer to the relevant description above, and will not repeat them here.

As yet another possible implementation manner, as shown in FIG. 22 , at S2201, the electronic device may acquire a first gesture performed by the user on the first area on the electronic device. Next, at S2202, the electronic device may control the electronic device according to the first gesture acquired at this time. Next, at S2203, the electronic device may acquire the eye movement type of the user. At S2204, when it is determined that the user's eye movement type is the first type (for example: jumping), the electronic device may control the electronic device according to the current eye movement type, and restrict the electronic device from acquiring the user's gestures. At S2205, the electronic device may continue to acquire the eye movement type of the user. In S2206, when it is determined that the user's eye movement type is switched from the first type (for example: jumping) to the second type (for example: non-beating), and the user's gaze position is switched from the first area on the electronic device to the second area , the electronic device can instruct the user to perform gesture control. Next, at S2207, the electronic device may acquire a second gesture performed by the user on the second area, and at S2208 control the electronic device according to the acquired second gesture. It can be understood that, for the process of obtaining the eye movement type and gesture of the electronic device, as well as the process of instructing the user to perform gesture control and controlling the electronic device, etc., please refer to the relevant description above for details, and will not go into details here. . Exemplarily, the process described in FIG. 22 can be understood as that the user first performs gesture control on an area on the electronic device, and then shifts the line of sight to another area on the electronic device, and then performs gesture control on the new area. . For example, as shown in Figure 24, in (B) to (D) of Figure 24, the user performs gesture control to the content in area 53; Then, switch from (D) of Figure 24 to (E) of Figure 24 During the process, the user's eye movement type acquired by the electronic device is jerk, at this time, the electronic device can determine the gaze position of the user's eyes in real time (that is, control the electronic device), and the electronic device can no longer obtain the user's gestures during this process. , or continue to acquire the user's gestures but do not respond (that is, restrict the electronic device from acquiring the user's gestures); then, as shown in (E) of Figure 24, the electronic device continues to acquire the user's eye movement type, and at this time the user's eye movement The type is switched from beating to non-jumping, and the user's gaze position is also switched from the position in (D) of Figure 24 to the position where the picture 63 is located. At this time, as shown in (F) of Figure 24, the electronic device can indicate the user Gesture control is performed, and the gestures performed by the user on the area where the picture 63 is located are subsequently obtained, and the electronic device is controlled according to the obtained gestures.

The above is the introduction of the technical solution provided by the present application, and the following sub-scenes are illustrated for ease of understanding.

scene one

In this scenario, multiple items are displayed on the display screen of the electronic device 100, such as applications, files, pictures, videos and so on.

As shown in (A) of Figure 23, user A can browse items on the electronic device 100, and during user A's browsing process, the eye movement tracking device matched with the electronic device 100 can determine that the eye movement type of user A's eyes is At this time, continue to judge the eye movement type of user A's eyes.

Next, as shown in (B) of FIG. 23 , user A's eyes start to fixate on item 61 , and the eye-tracking device on the electronic device 100 can determine that the eye-movement type of user A's eyes is not jitter. Afterwards, the gaze position of the user A's eyes can be determined by using the eye tracking device, and then the gaze area 53 can be determined, and the cursor 92 can be displayed at the lower left corner of the gaze area 53 . At this time, the user A can use his hand 22 to perform gesture operations in the operation area 52 corresponding to the gaze area 53 . But at this point User A changes his mind and rebrowses the item again. At this time, the eye-tracking device can be used to determine that the eye movement type of user A's eyes is twitching. At this time, the cursor 902 is no longer displayed, that is, the cursor 902 is controlled to disappear, and the electronic device shown in (C) of FIG. 23 is displayed. 100 displayed interface. In one example, in (B) of FIG. 23 , after determining that user A's eyes are fixed on item 61, in order to improve control efficiency, cursor 92 can be placed on item 61, so that user A can directly select item 61, and The item 61 can be selected without moving the cursor 92 .

Next, as shown in (D) of FIG. 23 , the user A's eyes start to focus on the item 62 . At this time, the eye-tracking device on the electronic device 100 can determine that the eye movement type of the user A's eyes is not jitter. After that, the gaze position of user A's eyes can be determined by using the eye tracking device matched with the electronic device 100 , and then the gaze area 53 can be determined, and the cursor 92 can be displayed at the lower left corner of the gaze area 53 . At this time, user A can use his hand 22 to perform gesture operations in the operation area 52 corresponding to the gaze area 53 . In one example, in (D) of FIG. 23 , after determining that the user A's eyes are fixed on the item 62, in order to improve the control efficiency, the cursor 92 can be placed on the item 62, so that the user A can directly select the item 62, and It is not necessary to move the cursor 92 to select the item 62 .

Next, the user A moves his hand 22 as shown in (E) of FIG. 23 . When the hand 22 of the user A moves, the gesture tracking device matched with the electronic device 100 can track the movement of the hand 22 . Afterwards, the electronic device 100 can control the cursor 92 in its gaze area 53 to move along with the hand 22 . When the cursor 92 moves to the item 62, the user A can complete the "confirmation" operation through the confirmation gesture. So far, the operation of selecting item 62 is completed.

In this way, by combining eye tracking and gesture tracking, users can complete item selection naturally and efficiently without using peripheral devices, and the system allows users to change their minds midway and complete secondary or multiple selections. On the other hand, since the eye movement control completes the rough selection for the gesture control, the distance and time of the user's hand movement during the entire control process will be very short, so the degree of fatigue is very low, which improves the user experience and control accuracy.

scene two

In this scenario, content such as text may be displayed on the electronic device 100 , and the user may read the text on the electronic device 100 at this time. Exemplarily, the scene may be a meeting, education and other scenes.

As shown in (A) of FIG. 24 , the user A reads the text being displayed on the display screen of the electronic device 100 . During user A's reading process, the eye movement tracking device matched with the electronic device 100 detects that the eye movement type is smooth tracking.

Next, as shown in (B) of FIG. 24 , the gaze position of user A's eyes can be determined by using the eye tracking device, and then the gaze area 53 can be determined, that is, the eye movement type of the user's eyes is not jumping, and at this time it can be A cursor 92 is displayed within the gaze area 53 . In order to prevent the cursor 92 from affecting user A's reading, the cursor 92 may appear in a place that does not block the reading line of sight. Wherein, the operation area 52 corresponding to the gaze area can also be determined in (B) of FIG. 24 , and the hand 22 of the user A can make a corresponding gesture operation in the operation area 52 .

Next, the user A moves his hand 22 as shown in (C) of FIG. 24 . When the hand 22 of the user A moves, the gesture tracking device matched with the electronic device 100 can track the movement of the hand 22 . Afterwards, the electronic device 100 can control the cursor 92 in its gaze area 53 to move along with the hand 22 . When the cursor 92 moves to the position where user A wants to mark, user A can stop moving his hand 22 and make a gesture to start marking, so as to mark.

Then, as shown in (D) of FIG. 24 , after user A makes the gesture to start labeling, he can continue to move his hand 22, and when he moves to the position where he wants to end labeling, he can make a gesture to stop labeling. Gesture, the labeling is ended at this moment, and the labeling area 531 is obtained at this moment. Wherein, during the process of user A marking, the cursor 92 may move along with user A's hand 22 . It can be understood that, in addition to the marking method described in (D) of FIG. 24 , user A can also use other marking methods when marking, such as marking with patterns, etc., which are not limited here.

Next, as shown in (E) of FIG. 24 , after user A finishes marking, user A's line of sight shifts to picture 63. At this time, the eye movement tracking device matched with electronic device 100 can detect user A's eye movement. The type is bouncing, so the cursor 92 can be controlled to disappear.

Next, as shown in (F) of FIG. 24 , after user A shifts his gaze to picture 63, he is attracted by picture 63. At this time, the eye movement tracking device matched with electronic device 100 can detect that user A's eye movement type is different. Therefore, the display of the cursor 92 can be controlled at this time, for example, the cursor 92 can be displayed in the gaze area 53 to instruct the user A to perform gesture control. In addition, in (F) of FIG. 24 , the operation area 52 corresponding to the gaze area can also be determined, and the user A's hand 22 can perform corresponding gesture operations in the operation area 52 .

Next, as shown in (G) of FIG. 24 , user A wants to mark text on the picture 63 , at this time, user A can move the cursor 92 by moving his hand 22 . When the cursor 92 is moved to the desired position of the user A, the user A can use a text writing gesture to write text in the manipulation area 52 , for example, write "smile" and so on. In the process of user A's writing, the gesture tracking device on the electronic device 100 can track the writing track of user A's hand 22, and then the display screen of the electronic device 100 can display the same content as the user's writing track. , for example to display "smile". In order to prevent user A from extending his hand 22 to the right during the writing process, after user A writes a certain number of words, user A can use a specific gesture to switch the writing position to the left side of the operation area 52, and then continue writing.

In this way, by combining eye movement and gesture control, users can naturally and efficiently complete remote annotation on electronic devices. Wherein, the type of annotation includes but not limited to pattern, text, symbol and so on. In addition, in the current related technologies, when marking, it is often done by close touch marking or remote marking by voice, but close touch marking requires users to approach the screen every time they mark, which is very inconvenient; and voice remote marking on the one hand Only text information can be input, and it is difficult to input pattern information, and speech signal recognition is also affected by environmental noise. However, in this solution, users can complete natural and efficient annotation of various information remotely, which improves the efficiency of annotation and user experience.

scene three

The scene may be an entertainment scene. For example, the scene may be a game scene. At this time, the user may be experiencing first-person shooting games, somatosensory games, puzzle games, chess and card games, and the like. The following uses a first-person shooter game as an example to introduce this scenario.

As shown in (A) of FIG. 25 , during the game of user A, when user A aims at the target 71 , the eye movement tracking device matched with the electronic device 100 detects that the eye movement type is gaze. Then, the gaze position of the user A's eyes can be determined by using the eye tracking device, and then the gaze area 53 can be determined, and the cursor 92 for aiming can be displayed in the gaze area 53 . Afterwards, the user A's hand 22 can make a shooting gesture in the operation area 52 corresponding to the gaze area 53 to complete the operation of shooting the target 71 .

Next, as shown in (B) of FIG. 25 , after the user A finishes shooting the target 71 , the user A finds the target 72 and shifts his gaze to the target 72 . During this process, the eye movement tracking device matched with the electronic device 100 detects that the eye movement type is jitter, and at this time, the cursor 92 can be controlled to disappear.

Next, as shown in (C) of FIG. 25 , when the user A aims at the target 72 , the eye movement tracking device matched with the electronic device 100 detects that the eye movement type is gaze. A cursor 92 for aiming will then appear within the gaze area 53 . At this time, the cursor 92 is not completely on the target 72 , in order to improve the shooting accuracy, the user A can adjust the cursor 92 . Wherein, as shown in (D) of Figure 25, the hand 22 of user A can make a movement gesture in the operation area 52 corresponding to the gaze area 53, and at this time, the cursor 92 can follow the hand 22 of user A to move . After aiming is further completed, the user A's hand 22 can make a shooting gesture in the operation area 52 corresponding to the gaze area 53 to complete the operation of shooting the target 72 .

In this way, by combining eye movement and gesture control, users can naturally and efficiently complete target switching in entertainment scenarios, and can also achieve higher-precision operations, thereby obtaining a smooth entertainment experience. In addition, the interaction mode of somatosensory games in the related art often requires the user to make relatively large movements, so the efficiency is low, fatigue is easy and the precision is low. In this solution, the user only needs to move the hand in a small range to complete the operation, so it is not easy to cause fatigue, and the control efficiency and precision are higher.

scene four

This scene can be a usage scene of a virtual large-screen interface created by electronic devices such as virtual reality (virtual reality, VR), augmented reality (augmented reality, AR), and mixed reality (mixed reality, MR).

As shown in (A) of FIG. 26 , multiple items are displayed on the field of view (FOV) of the virtual large-screen interface created by the electronic device 100, and user A is browsing items, such as item 64 and item 65. During user A's browsing process, the eye movement tracking device matched with the electronic device 100 can detect the eye movement type of user A's eyes. Cursor 92 is not displayed within the FOV at this time.

Next, as shown in (B) of FIG. 26 , when the user A focuses on the item 64 , the eye-tracking device matched with the electronic device 100 can determine that the eye movement type of the user A is not jitter. Afterwards, the eye tracking device can be used to determine the gaze position of the user A's eyes, and then the gaze area 53 can be determined, and a cursor 92 is displayed in the gaze area 53 to instruct user A to perform gesture control. At this time, the user A can use his hand 26 to perform gesture operations in the operation area 52 corresponding to the gaze area 53 .

Next, as shown in (C) of FIG. 26 , the user A wants to close the item 64 , and the user A can move his hand 26 in the operation area 52 . When the hand 26 of the user A moves, the gesture tracking device matched with the electronic device 100 can track the movement of the hand 26 . Afterwards, the electronic device 100 can control the cursor 92 in its gaze area 53 to move along with the hand 26 . User A may stop moving hand 26 when cursor 92 is moved over item 62 .

Next, as shown in (D) of FIG. 26 , user A can make a confirmation gesture with his hand 26 to complete the "confirmation" operation. Afterwards, the gesture tracking device matched with the electronic device 100 can track the “confirm” operation of the hand 26 . Then, the electronic device 100 can close the item 64 on its FOV. So far, the operation of closing item 64 is completed.

In this way, by combining eye movement and gesture control, users can achieve precise control of the VR/AR/MR interface through a small range of gestures. In addition, in the current related technologies, most VR/AR/MR devices use cameras to capture gestures for manipulation. On the one hand, the hand must complete the operation within the camera's field of view, which limits the degree of freedom of manipulation. On the other hand, the manipulation accuracy is also low. To meet the needs of users, eye movement control is used in this solution to select the gaze area first, which provides a starting point and scale for gesture control, and solves the problem of correspondence between non-line-of-sight gesture control and FOV interface coordinates, allowing users to efficiently and accurately Complete the interface control of VR/AR/MR equipment and improve the user experience.

scene five

The scene may be an entertainment scene. For example, the scene may be a scene of watching a video (such as watching a movie, etc.). In this scenario, a preset area is set on the electronic device. When the user looks at the preset area, it can be judged that the user currently has the intention to control the electronic device. When the user looks at other areas, it can be judged that the user does not currently exist. Intent to control electronic devices. It is understandable that when a user watches a video, the eye movement type of the user's eyes is not jumping in most cases. Therefore, such a design method can avoid misidentification and improve user experience.

Specifically, as shown in (A) of FIG. 27 , user A is watching a movie on the electronic device 100 . When user A is watching a movie, the eye movement tracking device matched with the electronic device 100 can detect the user's eye movement type in real time or periodically.

As shown in (B) of FIG. 27 , the user A is looking at the area 56 at this time. The area 56 is a preset area on the electronic device 100 . When the user gazes at the area 56, in (B) of FIG. 27 , the eye-tracking device matched with the electronic device 100 can detect that the eye movement type of user A is not jumping, and determine the gaze position of user A's eyes is the area 56, and turn to (C) of FIG. 27 at this time.

Next, as shown in (C) of FIG. 27 , the eye tracking device can be used to determine the gaze position of the user A's eyes, and then the gaze area 53 can be determined, and the playback progress and the cursor 92 can be displayed in the gaze area 53, At this time, the cursor 92 may be displayed in the lower left corner of the gaze area 53 . Wherein, the operation area 52 corresponding to the gaze area 53 can also be determined in (C) of FIG. 27 , and the hand 22 of the user A can perform corresponding gesture operations in the operation area 52 . It can be understood that when a user watches a video, he generally wants to adjust the progress or volume, etc. Therefore, in order to prevent the cursor 92 from affecting the viewing experience, the cursor 92 can be displayed at the position where the progress is adjusted or the volume is adjusted. In addition, considering that the progress adjustment and the volume adjustment are generally not in the same position, the volume adjustment key and the playback progress bar can be displayed at the same time at this time, so that the user can make a selection based on needs. Of course, the electronic device 100 can also be provided with an area corresponding to the volume adjustment and an area corresponding to the playback progress. When the user looks at the area corresponding to the volume adjustment, the volume adjustment key will be displayed. When the user looks at the area corresponding to the playback progress area, the playback progress bar will be displayed. In other words, a target area corresponding to the target function may be set on the electronic device 100 , and when the user looks at the target area, control keys related to the target function will be displayed on the electronic device.

Next, the user A moves his hand 22 as shown in (D) of FIG. 27 . When the hand 22 of the user A moves, the gesture tracking device matched with the electronic device 100 can track the movement of the hand 22 . Afterwards, the electronic device 100 can control the cursor 92 in its gaze area 53 to move along with the hand 22 . When the cursor 92 moves to a position that user A wants to adjust, user A can stop moving his hand 22 .

Next, as shown in (E) of FIG. 27 , user A can make a gesture of selecting the progress bar and drag the progress bar. During this process, the gesture tracking device matched with the electronic device 100 can track the movement of the hand 22 . Afterwards, the electronic device 100 can control the cursor 92 in its gaze area 53 to move along with the hand 22 . When the cursor 92 moves to the position that user A wants to mark, user A can stop moving his hand 22 .

Next, as shown in (F) of FIG. 27 , after user A stops dragging the progress bar, the eye-tracking device matched with the electronic device 100 can detect that the eye movement type of user A is jumping, so the cursor 92 can be controlled to disappear .

In this way, by combining eye movement control and gesture control, there can be no sense interaction between eye movement control and gesture control, so that users can naturally and efficiently complete the adjustment of playback progress, sound, etc. in entertainment scenes, and can also achieve higher precision. operation, so as to obtain high-quality control experience and movie viewing experience.

It can be understood that the processor in the embodiments of the present application may be a central processing unit (central processing unit, CPU), and may also be other general processors, digital signal processors (digital signal processor, DSP), application specific integrated circuits (application specific integrated circuit, ASIC), field programmable gate array (field programmable gate array, FPGA) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. A general-purpose processor can be a microprocessor, or any conventional processor.

The method steps in the embodiments of the present application may be implemented by means of hardware, or may be implemented by means of a processor executing software instructions. The software instructions can be composed of corresponding software modules, and the software modules can be stored in random access memory (random access memory, RAM), flash memory, read-only memory (read-only memory, ROM), programmable read-only memory (programmable rom) , PROM), erasable programmable read-only memory (erasable PROM, EPROM), electrically erasable programmable read-only memory (electrically EPROM, EEPROM), register, hard disk, mobile hard disk, CD-ROM or known in the art any other form of storage medium. An exemplary storage medium is coupled to the processor such the processor can read information from, and write information to, the storage medium. Of course, the storage medium may also be a component of the processor. The processor and storage medium can be located in the ASIC.

In the above embodiments, all or part of them may be implemented by software, hardware, firmware or any combination thereof. When implemented using software, it may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on the computer, the processes or functions according to the embodiments of the present application will be generated in whole or in part. The computer can be a general purpose computer, a special purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in or transmitted via a computer-readable storage medium. The computer instructions may be transmitted from one website site, computer, server, or data center to another website site by wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) , computer, server or data center for transmission. The computer-readable storage medium may be any available medium that can be accessed by a computer, or a data storage device such as a server or a data center integrated with one or more available media. The available medium may be a magnetic medium (such as a floppy disk, a hard disk, or a magnetic tape), an optical medium (such as a DVD), or a semiconductor medium (such as a solid state disk (solid state disk, SSD)), etc.

It can be understood that the various numbers involved in the embodiments of the present application are only for convenience of description, and are not used to limit the scope of the embodiments of the present application.

Claims

A method for interacting with an electronic device, wherein the method includes:

The electronic device obtains the eye movement type of the user's eyes;

When it is determined that the eye movement type is the first type, the electronic device controls the electronic device according to the first type;

When determining that the eye movement type is the second type, the electronic device instructs the user to perform gesture control;

The electronic device obtains the user's gesture;

The electronic device controls the electronic device according to the acquired gesture.
The method according to claim 1, wherein the instructing the user to perform gesture control specifically comprises:

Obtain the gaze position of the user;

Prompting the user to perform gesture control within a first preset area where the gaze position is located.
The method according to claim 1 or 2, wherein the instructing the user to perform gesture control specifically comprises:

determining a target area according to the type of the first content displayed on the electronic device;

Prompting the user to perform gesture control in the target area, wherein the target area is an area on the electronic device that does not contain the first content.
The method according to any one of claims 1-3, wherein the instructing the user to perform gesture control specifically includes:

Obtain the gaze position of the user;

determining a gaze area of the user according to the gaze position;

When the gaze area is the second preset area, the user is prompted to perform gesture control.
The method according to any one of claims 1-4, wherein the instructing the user to perform gesture control includes one or more of the following:

Controlling the electronic device to display a cursor, controlling to enlarge at least part of the content displayed by the electronic device, or controlling a sound component matched with the electronic device to broadcast a first voice, the first voice is used to prompt the user to perform gesture control.
The method according to claim 5, wherein the instructing the user to perform gesture control comprises controlling the electronic device to display a cursor, and the cursor is displayed in a first area of the display area of the electronic device;

The method also includes:

determining that the eye movement type continues to be the second type, when the user's eyes move and the user's gaze area is switched from the first area to the second area in the display area of the electronic device , controlling the cursor to move from the first area to the second area, wherein the gaze area is determined by the gaze position of the user's eyes.
The method according to claim 5 or 6, wherein the instructing the user to perform gesture control includes controlling to enlarge at least part of the content displayed by the electronic device, and the enlarged content is the first in the display area of the electronic device. the content displayed in an area;

The method also includes:

determining that the eye movement type continues to be the second type, and when the user's eyes move and the user's gaze area is switched from the first area to the second area of the electronic device, the first The enlarged content in the area is restored to the original state, and the content in the second area is enlarged.
The method according to any one of claims 1-7, wherein after the electronic device instructs the user to perform gesture control, the method further comprises:

When it is determined that the eye movement type is switched from the second type to the first type, the electronic device stops instructing the user to perform gesture control, and/or, the electronic device performs gesture control according to the first type The electronic device controls and restricts the electronic device from acquiring the user's gesture.
The method according to any one of claims 1-8, further comprising: after determining that the eye movement type is the second type:

Obtain the gaze position of the user;

determining the size of the user's gaze area according to the gaze position;

According to the size of the gaze area, the size of the operation area when the user performs gesture control is determined.
The method according to any one of claims 1-9, wherein the electronic device controls the electronic device according to the first type, specifically comprising:

The electronic device determines the gaze position of the user in real time according to the first type, and/or, the electronic device controls switching of content displayed on the electronic device according to the first type.
A method for interacting with an electronic device, characterized in that the method includes:

The electronic device acquires a user's gesture;

The electronic device controls the electronic device according to the acquired gesture;

The electronic device obtains the eye movement type of the user;

When it is determined that the user's eye movement type is the first type, the electronic device controls the electronic device according to the first type, and restricts the electronic device from controlling the electronic device according to the user's gesture ;

When determining that the user's eye movement type is the second type, the electronic device acquires the user's gesture;

The electronic device controls the electronic device according to the acquired gesture.
The method according to claim 11, wherein after the electronic device controls the electronic device according to the first type and restricts the electronic device from acquiring the user's gesture, the method further comprises :

When it is determined that the eye movement type of the user is switched from the first type to the second type, the electronic device instructs the user to perform gesture control;

The electronic device obtains the user's gesture;

The electronic device controls the electronic device according to the acquired gesture.
The method according to claim 11 or 12, wherein the limiting the electronic device to control the electronic device according to the user's gesture specifically comprises: the electronic device does not acquire the user's gesture, or , although the electronic device acquires the gesture of the user, it does not process the gesture.
A method for interacting with an electronic device, characterized in that the method comprises:

The electronic device acquires a first gesture performed by the user on the first area on the electronic device;

The electronic device controls the electronic device according to the acquired first gesture;

The electronic device obtains the eye movement type of the user;

When it is determined that the user's eye movement type is the first type, the electronic device controls the electronic device according to the first type, and restricts the electronic device from acquiring gestures of the user;

The electronic device continues to obtain the eye movement type of the user;

When it is determined that the eye movement type of the user is switched from the first type to the second type, and the gaze position of the user is switched from the first area on the electronic device to the second area, the electronic the device instructs the user to perform gesture control;

acquiring, by the electronic device, a second gesture performed by the user on the operation of the second area;

The electronic device controls the electronic device according to the acquired second gesture.
A method for interacting with an electronic device, characterized in that the method comprises:

The electronic device obtains the user's first eye movement information;

The electronic device determines a first eye movement type according to the first eye movement information;

The electronic device controls the electronic device according to the first eye movement type;

The electronic device acquires the second eye movement information of the user;

The electronic device determines a second eye movement type according to the second eye movement information;

The electronic device obtains the user's gesture;

The electronic device controls the electronic device according to the acquired gesture.
method according to claim 15, is characterized in that, described method also comprises:

The electronic device obtains the user's third eye movement information;

The electronic device determines the first eye movement type according to the third eye movement information;

The electronic device controls the electronic device according to the first eye movement type, and restricts the electronic device from acquiring gestures of the user.
An electronic device, characterized in that it comprises:

An eye-tracking device for obtaining user's eye-movement information;

a gesture tracking device, configured to acquire gestures of the user;

at least one memory for storing programs;

At least one processor configured to execute the program stored in the memory, when the program stored in the memory is executed, the processor is configured to:

controlling the eye-tracking device to acquire eye-movement information of the user;

When the eye movement type of the user's eyes determined by the eye movement information is a first type, control the electronic device according to the first type;

When the eye movement type of the user's eyes determined by the eye movement information is the second type, controlling the electronic device to instruct the user to perform gesture control;

controlling the gesture tracking device to acquire gestures of the user;

The electronic device is controlled according to the gesture acquired by the gesture tracking device.
The electronic device according to claim 17, wherein the processor controls the electronic device to instruct the user to perform gesture control, specifically comprising:

The processor determines the user's gaze position according to the user's eye movement information;

The processor controls the electronic device to prompt the user to perform gesture control within a first preset area where the gaze position is located.
The electronic device according to claim 17 or 18, wherein the processor controls the electronic device to instruct the user to perform gesture control, specifically comprising:

The processor determines a target area according to the type of the first content displayed on the electronic device;

The processor controls the electronic device to prompt the user to perform gesture control in the target area, wherein the target area is an area on the electronic device that does not contain the first content.
The electronic device according to any one of claims 17-19, wherein the processor controls the electronic device to instruct the user to perform gesture control, specifically including:

The processor obtains the user's gaze position according to the user's eye movement information;

The processor determines the gaze area of the user according to the gaze position;

When the gaze area is the second preset area, the processor controls the electronic device to prompt the user to perform gesture control.
The electronic device according to any one of claims 17-20, wherein the processor controls the electronic device to instruct the user to perform gesture control, including one or more of the following:

The processor controls the electronic device to display a cursor, the processor controls to amplify at least part of the content displayed by the electronic device, or the processor controls a sound component matched with the electronic device to broadcast the first voice, The first voice is used to prompt the user to perform gesture control.
The electronic device according to claim 21, wherein the processor controlling the electronic device to instruct the user to perform gesture control comprises the processor controlling the electronic device to display a cursor, and the cursor is displayed on the in the first area of the display area of the electronic device;

The processor is also used to:

When the eye movement type of the user's eyes determined by the eye movement information continues to be the second type, and the user's eyes move and the user's gaze area is determined by the display area of the electronic device When the first area is switched to the second area, control the electronic device to move the cursor from the first area to the second area, wherein the gaze area is determined by the gaze position of the user's eyes .
The electronic device according to claim 21 or 22, wherein the processor controlling the electronic device to instruct the user to perform gesture control comprises controlling the processor to enlarge at least part of the content displayed by the electronic device, And the enlarged content is the content displayed in the first area of the display area of the electronic device;

The processor is also used to:

When the eye movement type of the user's eyes determined by the eye movement information continues to be the second type, and the user's eyes move and the user's gaze area is switched from the first area of the electronic device to In the second area, the electronic device is controlled to restore the enlarged content in the first area to an initial state, and to enlarge the content in the second area.
The electronic device according to any one of claims 17-23, wherein after the processor controls the electronic device to instruct the user to perform gesture control, the processor is further configured to:

When the eye movement type of the user's eyes determined by the eye movement information is switched from the second type to the first type, control the electronic device to stop instructing the user to perform gesture control, and/or, according to the The first type controls the electronic device, and restricts the gesture tracking device from acquiring the user's gestures.
The electronic device according to any one of claims 17-24, wherein after the eye movement type of the user's eyes determined by the eye movement information is the second type, the processor is further configured to:

Obtain the gaze position of the user;

determining the size of the user's gaze area according to the gaze position;

According to the size of the gaze area, the size of the operation area when the user performs gesture control is determined.
The electronic device according to any one of claims 17-25, wherein the processor controls the electronic device according to the first type, specifically comprising:

The processor determines the gaze position of the user in real time according to the first type, and/or, the processor controls the electronic device to switch the content displayed on the electronic device according to the first type .
An electronic device, characterized in that it comprises:

An eye-tracking device for obtaining user's eye-movement information;

a gesture tracking device, configured to acquire gestures of the user;

at least one memory for storing programs;

At least one processor configured to execute the program stored in the memory, when the program stored in the memory is executed, the processor is configured to:

controlling the user's gesture acquired by the gesture tracking device;

controlling the electronic device according to the gesture acquired by the gesture tracking device;

When the eye movement type of the user's eyes determined by the eye movement information is the first type, control the electronic device according to the first type, and limit the control of the electronic device according to the user's gesture;

When the eye movement type of the user's eyes determined by the eye movement information is the second type, the electronic device is controlled according to the gesture acquired by the gesture tracking device.
The electronic device according to claim 27, wherein the processor further uses At:

When the eye movement type of the user's eyes determined by the eye movement information is switched from the first type to the second type, controlling the electronic device to instruct the user to perform gesture control;

controlling the user's gesture acquired by the gesture tracking device;

The electronic device is controlled according to the gesture acquired by the gesture tracking device.
The electronic device according to claim 27 or 28, wherein the processor limits the control of the electronic device according to the user's gesture, specifically comprising:

The processor controls the gesture tracking device not to acquire the user's gesture, or the processor controls the gesture tracking device to continue acquiring the user's gesture but does not process the gesture.
An electronic device, characterized in that it comprises:

An eye-tracking device for obtaining user's eye-movement information;

a gesture tracking device, configured to acquire gestures of the user;

at least one memory for storing programs;

At least one processor configured to execute the program stored in the memory, when the program stored in the memory is executed, the processor is configured to:

controlling the gesture tracking device to acquire gestures of the user;

controlling the electronic device according to the first gesture of the user operating on the first area on the electronic device acquired by the gesture tracking device;

controlling the eye-tracking device to acquire eye-movement information of the user;

When the eye movement type of the user's eyes determined by the eye movement information is the first type, control the electronic device according to the first type, and restrict the gesture tracking device from acquiring the user's gesture;

continue to control the eye-tracking device to obtain eye-movement information of the user;

When the eye movement type of the user's eyes determined by the eye movement information is switched from the first type to the second type, and the gaze position of the user is switched from the first area on the electronic device to the second area, controlling the electronic device to instruct the user to perform gesture control;

controlling the gesture tracking device to acquire gestures of the user;

The electronic device is controlled according to the second gesture of the user operating on the second area acquired by the gesture tracking device.
An electronic device, characterized in that it comprises:

An eye-tracking device for obtaining user's eye-movement information;

a gesture tracking device, configured to acquire gestures of the user;

at least one memory for storing programs;

At least one processor configured to execute the program stored in the memory, when the program stored in the memory is executed, the processor is configured to:

controlling the eye-tracking device to acquire first eye-movement information of the user;

determining a first eye movement type according to the first eye movement information;

controlling the electronic device according to the first eye movement type;

controlling the eye-tracking device to obtain second eye-movement information of the user;

determining a second eye movement type according to the second eye movement information;

controlling the gesture tracking device to acquire gestures of the user;

The electronic device is controlled according to the gesture acquired by the gesture tracking device.
The electronic device according to claim 31, wherein the processor is also used for:

controlling the eye-tracking device to acquire third eye-movement information of the user;

determining the first eye movement type according to the third eye movement information;

The electronic device is controlled according to the first eye movement type, and the gesture tracking device is restricted from acquiring gestures of the user.
A computer-readable storage medium, the computer-readable storage medium stores a computer program, and when the computer program is run on an electronic device, the electronic device executes the method according to any one of claims 1-16 .
A computer program product, characterized in that, when the computer program product is run on an electronic device, the electronic device is made to execute the method according to any one of claims 1-16.