WO2019214442A1

WO2019214442A1 - Device control method, apparatus, control device and storage medium

Info

Publication number: WO2019214442A1
Application number: PCT/CN2019/084230
Authority: WO
Inventors: 秦林婵; 刘琳; 黄通兵
Original assignee: 北京七鑫易维信息技术有限公司
Priority date: 2018-05-11
Filing date: 2019-04-25
Publication date: 2019-11-14
Also published as: CN108681399B; CN108681399A

Abstract

The present application discloses a device control method, an apparatus, a control device and a storage medium Said method comprises: when a user looks at a first device, determining an identity identifier of the first device according to image information of the first device, and acquiring an eye image and motion information of the user; determining, according to a point-of-regard corresponding to the eye image and the motion information, an interactive operation instruction to be executed; when the user looks at a second device, determining an identity identifier of the second device according to the image information of the second device; and on the basis of the interactive operation instruction, sending the identity identifier of the first device, the identity identifier of the second device, and the interactive operation instruction to the first device and/or the second device.

Description

Device control method, device, control device and storage medium

Technical field

The embodiments of the present invention relate to the field of Sight Tracking technologies, and in particular, to a device control method, apparatus, control device, and storage medium.

Background technique

Human-computer interaction refers to the process of information exchange between a person and a machine in which a person and a machine (such as a computer) use a certain dialogue language in a certain interaction manner to complete the task. With the development of related fields such as machine learning and computer vision, machines can recognize gestures, gestures, and lines of sight of users. Among them, the line of sight has gradually become one of the most important interactive means of human-computer interaction, and has broad application prospects in the field of human-computer interaction.

Sight line tracking is to capture the user's eye image by using the eye movement measuring device, estimate the user's line of sight direction by analyzing the feature information in the eye image, and then predict the user's demand based on the user's line of sight direction, and respond to achieve the purpose of the eye control device.

At present, line-of-sight tracking is widely used to control devices, but does not control the interaction process between devices.

Summary of the invention

The device control method, device, control device and storage medium provided by the application can effectively control the interaction between devices.

In a first aspect, an embodiment of the present application provides a device control method, including:

Determining an identity of the first device according to image information of the first device, and acquiring an eye image and motion information of the user, when the user looks at the first device;

Determining an interworking instruction to be executed according to the gaze point corresponding to the eye image and the motion information;

Determining an identity of the second device according to image information of the second device when the user is watching the second device;

Sending, according to the interaction operation instruction, the identity identifier of the first device, the identity identifier of the second device, and the interaction operation instruction to the first device and/or the second device, so as to The first device and/or the second device execute the interworking instruction.

In a second aspect, the embodiment of the present application further provides a device control apparatus, including:

An information acquiring module, configured to determine an identity of the first device according to image information of the first device, and obtain an eye image and motion information of the user, when the user looks at the first device;

An instruction determining module, configured to determine an interaction operation instruction to be executed according to the gaze point corresponding to the eye image and the motion information;

An identifier determining module, configured to determine, according to image information of the second device, an identity identifier of the second device when the user looks at the second device;

a sending module, configured to send, according to the interaction operation instruction, an identity of the first device, an identity of the second device, and the interaction operation instruction to the first device and/or the second a device to cause the first device and/or the second device to execute the interworking instruction.

In a third aspect, the embodiment of the present application further provides a control device, including:

One or more processors;

a storage device for storing one or more programs;

The one or more programs are executed by the one or more processors, such that the one or more processors implement the device control method provided by the embodiments of the present application.

In a fourth aspect, the embodiment of the present application further provides a computer readable storage medium, where the computer program is stored, and when the program is executed by the processor, the device control method provided by the embodiment of the present application is implemented.

An embodiment of the present application provides a device control method, apparatus, control device, and storage medium. First, when a user looks at a first device, the identity identifier of the first device is determined according to image information of the first device, and is obtained. a user's eye image and motion information; secondly, determining an interoperation instruction to be executed according to the gaze point corresponding to the eye image and the motion information; and then, when the user looks at the second device, according to the second device The image information determines the identity of the second device; finally, based on the interaction operation instruction, sending the identity of the first device, the identity of the second device, and the interaction operation instruction to the first a device and/or the second device to cause the first device and/or the second device to execute the interworking instruction. With the above technical solution, an interoperation instruction to be executed between the devices can be determined based on the eye image and the motion information acquired by the user looking at the first device, to identify the identity of the first device and the identity of the second device based on the interaction operation instruction. The identification and interworking instructions are sent to the first device and/or the second device to cause the first device and/or the second device to perform the interworking instructions to complete the interaction between the devices. In the process of controlling the device, the user's eye point is analyzed to determine the user's gaze point and the user's motion information is effectively implemented to control the interaction between multiple devices, and the application scenario of the line-of-sight tracking technology is extended. .

DRAWINGS

1 is a schematic flowchart of a device control method according to Embodiment 1 of the present application;

2a is a schematic flowchart of a device control method according to Embodiment 2 of the present application;

FIG. 2b is a schematic diagram of an eye image provided by the second embodiment;

FIG. 2 is a schematic diagram of an application scenario of a device control method according to Embodiment 2 of the present application;

3 is a schematic structural diagram of a device control apparatus according to Embodiment 3 of the present application;

FIG. 4 is a schematic structural diagram of a control device according to Embodiment 4 of the present application.

detailed description

The present application will be further described in detail below with reference to the accompanying drawings and embodiments. It is understood that the specific embodiments described herein are merely illustrative of the application and are not intended to be limiting. In addition, it should be noted that, for the convenience of description, only some but not all of the structures related to the present application are shown in the drawings.

Before discussing the exemplary embodiments in more detail, it should be noted that some exemplary embodiments are described as a process or method depicted as a flowchart. Although the flowcharts describe various operations (or steps) as a sequential process, many of the operations can be implemented in parallel, concurrently or concurrently. In addition, the order of operations can be rearranged. The process may be terminated when its operation is completed, but may also have additional steps not included in the figures. The processing may correspond to methods, functions, procedures, subroutines, subroutines, and the like.

Embodiment 1

FIG. 1 is a schematic flowchart of a device control method according to Embodiment 1 of the present application. The method is applicable to a device that controls a plurality of devices to interact through a Sight Tracking technology. The method may be implemented by the device control device provided by the embodiment of the present application. Execution, wherein the device can be implemented by software and/or hardware and is generally integrated on the control device. The control device in this embodiment includes, but is not limited to, an eye tracking device.

As shown in FIG. 1 , a device control method provided in Embodiment 1 of the present application includes the following steps:

S101. When the user looks at the first device, determine an identity of the first device according to image information of the first device, and obtain an eye image and motion information of the user.

In this embodiment, the user can be understood as a person who uses the control device to perform interactive control between devices. The first device can be understood as the device that the user is going to control. The image information can be understood as an image of the first device taken by the image capture device. An identity can be understood as identity information that uniquely identifies a device identity, such as a device ID. Motion information can be understood as information generated by user motion, such as head movements, eye movements, and/or gesture movements.

Generally, when the user looks at the first device, the step may determine the identity of the first device according to the image information of the first device, so as to implement interaction between the devices based on the determined identity. When determining the identity of the first device based on the image information of the first device, the step may perform image recognition on the image information to determine identifier information that can identify the device in the image, and determine an identity identifier of the device based on the identifier information; It is also possible to preset a preset device database. The image information of each device and the identity of each device may be pre-stored in the preset device database. After the image information of the first device is determined, the preset device database may be searched to determine the identity of the first device.

In addition, when the user looks at the first device, the step may further acquire an eye image and motion information of the user for the first device by using the collecting device. In this step, the eye image and the motion information of the user for the first device may be acquired by the image capturing device, wherein the motion information may also be determined by the motion information collecting device, such as a displacement sensor. It should be noted that the image acquisition device that acquires the first device and the image acquisition device that acquires the image of the user's eye may be the same device or different devices.

It should be noted that, when the user looks at the first device, the step may determine the identity of the first device according to the first device image information, and may also acquire the eye image and the motion information of the user. The order in which the identification and acquisition of the eye image and motion information are performed is not defined here. It is only necessary to determine the identity of the first device when acquiring the eye image and motion information of the user for the first device.

It can be understood that, in this step, when the user looks at the first device, the user's eye image, the motion information, and the image information of the first device may be acquired, and when the interactive operation instruction is determined, it is determined according to the first device. The image information determines the identity of the first device.

It can be understood that before the user controls the interaction between the devices, the user can first monitor whether there is a startup command. When the startup command is detected, the user can be considered to control the device at this time. Then, this step can determine whether the first device can be monitored by the image collecting device, where "first" is used only for distinguishing the devices. When the first device is detected, it can be considered that the user is watching the first device at this time.

Further, the motion information includes: head motion information and/or gesture information. The obtaining of the head motion information and the gesture information may be determined by the motion information collecting device in the control device, or may be determined by analyzing the image of the user at each moment, and the image may cover the range of motion of the user's head and the hand.

S102. Determine an interworking instruction to be executed according to the gaze point corresponding to the eye image and the motion information.

In this embodiment, the gaze point can be understood as the gaze location of the user on the first device. The interworking instructions to be executed can be understood as control commands for inter-device interactions to be performed.

After acquiring the eye image and the motion information of the user, the step may determine an interaction operation instruction to be executed based on the motion information and the gaze point corresponding to the eye image. The gaze point, the motion information, and the interoperation instruction corresponding to the user's eye image may have a one-to-one correspondence, and the corresponding interoperation instruction can be uniquely determined by the gaze point and the motion information corresponding to the user's eye image.

It can be understood that, when determining the gaze point corresponding to the eye image, the step may be determined based on the corresponding relationship between the eye image and the gaze point established in advance; or the eye image may be identified by identifying the eye image. The line of sight direction of the image, which in turn determines the gaze point of the line of sight on the first device.

The corresponding relationship between the eye image and the gaze point may be obtained by training in advance to adapt to each user; or different correspondences may be provided for different users. If there is a different correspondence between the eye images and the gaze points of different users, the embodiment may perform identity authentication on the eyes by performing iris recognition on the eye images to determine a gaze point corresponding to the image of the user's eyes. .

S103. Determine, when the user looks at the second device, the identity of the second device according to the image information of the second device.

In this embodiment, the second device can be understood as a device to be interacted with the first device. The second device can have more than one device.

Generally, when the user looks at the second device, the step of acquiring the image information of the second device by the image collecting device determines the identity of the second device. The specific operation of determining the identity of the second device by using the image information of the second device in this step is similar to the operation of determining the identity of the first device by using the image information of the first device, and details are not described herein.

It can be understood that the image capturing device can acquire the device image in real time. When it is detected that the image information collected is not the image information of the first device, the user can be considered to be looking at the second device. In addition, this embodiment can also be used to identify that the user is looking at the second device at this time by setting a gaze command. The gaze command may be generated by a button, or may be obtained by analyzing specific motion information of the user or feature information in the eye image. If the user is nodding when it is detected, it can be considered that the user is looking at the second device at this time.

S104. Send, according to the interaction operation instruction, an identity identifier of the first device, an identity identifier of the second device, and the interaction operation instruction to the first device and/or the second device, to Causing the first device and/or the second device to execute the interworking instruction.

Generally, the interaction between the devices (such as the interaction between the first device and the second device) may have different interoperation instructions, and different interoperation instructions may have different operations. Therefore, the step may be configured to send, according to the determined interworking instruction, the identity of the first device, the identity of the second device, and the interworking instruction to the first device and/or the second device, so that the first device and/or Or the second device completes the interaction between the devices based on the interworking instructions and the corresponding identity.

Exemplarily, when the interworking instruction is to send a file, the step may send the interoperation instruction, the identity of the first device, and the identity of the second device to the first device, to indicate that the first device is based on the interaction. The instruction sends a file to the second device.

A device control method is provided in the first embodiment of the present application. When the user looks at the first device, the identity of the first device is determined according to the image information of the first device, and the eye image and motion information of the user are obtained. And determining an interoperation instruction to be executed according to the gaze point corresponding to the eye image and the motion information; and then determining, by the user, the second device according to image information of the second device; The identity of the first device, the identity of the second device, and the interactive operation instruction are sent to the first device and/or the second device based on the interaction instruction. a device to cause the first device and/or the second device to execute the interworking instruction. With the above method, an interoperation instruction to be executed between the devices can be determined based on the eye image and the motion information acquired by the user looking at the first device, to identify the identity of the first device and the identity of the second device based on the interaction operation instruction. And interworking instructions are sent to the first device and/or the second device to cause the first device and/or the second device to perform the interworking instruction to complete the interaction between the devices. In the process of controlling the device, the user's eye point is analyzed to determine the user's gaze point and the user's motion information is effectively implemented to control the interaction between multiple devices, and the application scenario of the line-of-sight tracking technology is extended. .

Embodiment 2

FIG. 2 is a schematic flowchart of a device control method according to Embodiment 2 of the present application. The second embodiment is optimized based on the foregoing embodiments. In this embodiment, determining the identity of the first device according to the image information of the first device, further implementing: acquiring image information of the first device; searching and searching in a preset device database The identifier information corresponding to the image information of the first device is used as the identity identifier of the first device.

Further, the embodiment further determines an interaction operation instruction to be executed according to the gaze point corresponding to the eye image and the motion information, and further optimizes to: extract feature information in the eye image, the feature information Having a pupil feature; determining a line of sight corresponding to the eye image according to the feature information; determining a gaze point of the user on the first device according to the determined line of sight; searching in an instruction database of the first device An instruction corresponding to the fixation point and the motion information is used as an interoperation instruction.

The identifier of the second device is determined according to the image information of the second device, and is specifically optimized to: acquire image information of the second device; and search for a location in the preset device database. The identifier information corresponding to the image information of the second device is used as the identity identifier of the second device.

Further, the embodiment further sends, according to the interaction operation instruction, the identity identifier of the first device, the identity identifier of the second device, and the interaction operation instruction to the first device and/or the The second device is specifically optimized to: select the first device and/or the second device as a target device according to the operation corresponding to the interaction operation instruction; and identify the identity of the first device, The identity of the two devices and the interworking instructions are sent to the target device. For the details of this embodiment, please refer to the first embodiment.

As shown in FIG. 2a, a device control method provided in Embodiment 2 of the present application includes the following steps:

S201. Acquire an eye image and motion information of the user when the user looks at the first device, and acquire image information of the first device.

In this embodiment, when the user looks at the first device, the embodiment may acquire the eye image and the motion information of the user through the image capturing device, and acquire image information of the first device to determine the image based on the eye image and the motion information. The interaction operation instruction determines the identity of the first device according to the image information of the first device.

S202. Search for the identifier information corresponding to the image information of the first device in the preset device database as the identity identifier of the first device.

In this embodiment, the preset device database can be understood as a pre-set database storing the identity of each device and the corresponding identification device.

After obtaining the image information of the first device, the step may be performed in the preset device database to search for the identifier information corresponding to the image information of the first device, and use the found identifier information as the identity of the first device. The identifier is such that the remaining devices can interact with the first device through the identity.

It can be understood that an image of each device may be stored in the preset device database; or may be feature information of each device, such as a color, a shape, and/or an identification code, which can uniquely identify the device.

S203. Extract feature information in the eye image, where the feature information includes a pupil feature.

In the present embodiment, the feature information can be understood as information in the eye image that can identify the eye line of sight.

After acquiring the eye image of the user, the step may identify the eye image to extract feature information in the eye image, so that the gaze point of the user may be determined based on the determined feature information. The pupil features include, but are not limited to, pupil edge features, pupil radius, and/or pupil center position.

Generally, when the image of the user's eye is acquired by the image capturing device, the image capturing device may be an ordinary camera or an infrared camera. When the image capturing device in this embodiment is an infrared camera, the infrared camera further needs to be configured with an infrared lamp for manufacturing a spot in the eyeball by the infrared lamp, so that the user's pupil feature can be collected to determine the user's line of sight to further determine. Corresponding gaze point.

S204. Determine a line of sight corresponding to the eye image according to the feature information.

When the embodiment uses an ordinary camera to acquire an image of the user's eye, this step may determine the line of sight of the user based on the pupil feature in the eye image. For example, in this step, the user's line of sight can be determined by the feature information and a pre-built feature and line of sight comparison table.

When the embodiment uses the infrared camera to acquire the image of the user's eye, this step may determine the user in the eye image by using the corneal reflection method according to the pupil feature in the eye image and the spot information displayed by the infrared light in the eye image. Sight.

Among them, when the cornea reflection method is used to determine the direction of the user's line of sight, the main hardware requirements include but are not limited to: light source: generally infrared light source, because infrared light does not affect the vision of the eye; and can be a plurality of infrared light sources, predetermined Arrangement, such as a font, a shape, etc.; image acquisition device: for example, an infrared camera device, an infrared image sensor, a camera or a video camera.

The process of determining the direction of the user's line of sight includes:

Firstly, the eye image is obtained: the light source is directed to the user's eyes, and the image capturing device is used to photograph the user's eyes, and the reflection point of the light source on the cornea, that is, the light spot (also referred to as a Puerchen spot), is obtained, thereby obtaining the spot with the light spot. Eye image. Then, the line of sight/gaze point estimation is performed: as the eyeball rotates, the relative positional relationship between the pupil center and the spot changes accordingly, and the corresponding acquired eye images with the spot reflect such positional change relationship; The position change relationship is evaluated for line of sight/gaze points.

In general, eye tracking, also known as gaze tracking, is a technique for estimating the line of sight and/or gaze of the eye by measuring the motion of the user's eyes. Currently widely used is the optical recording method: an image capturing device (such as a camera or a video camera) is used to record the eye movement of the subject, that is, obtain an eye image reflecting the movement of the eye, and extract the eye from the acquired eye image. Features are used to model the line of sight/gaze point estimates. Wherein, the eye features (feature information in the eye image) may include: a pupil position, a pupil shape, an iris position, an iris shape, an eyelid position, an eye corner position, and/or a spot (also referred to as a Purpillar spot) position.

It can be understood that when the user's line of sight direction is acquired, the contact/non-contact sensor (for example, an electrode, a capacitive sensor) can also be used to estimate the movement of the eye.

FIG. 2b is a schematic diagram of an eye image provided in the second embodiment, and FIG. 2b shows an eye, which is not limited to the number of eyes. The eye image acquired in this embodiment may also include two images. Eyes to improve the accuracy of the process. Specifically, in this embodiment, the feature information may be determined based on the acquired eye image to obtain a line of sight corresponding to the eye image. As shown in Fig. 2b, the present embodiment can obtain the user's line of sight through the pupil 21 and/or the iris 22 in the eye image. In addition, when the user's eye image is collected by using the infrared camera, the embodiment may also acquire the spot information included in the eye image to assist in obtaining the line of sight corresponding to the eye image.

S205. Determine, according to the determined line of sight, a gaze point of the user on the first device.

After determining the line of sight corresponding to the user's eye image, the step may determine the gaze point coordinate of the line of sight direction on the first device based on the determined line of sight to obtain the gaze point of the user on the first device. Among them, the line of sight can be understood as a three-dimensional vector, and the gaze point can be understood as the two-dimensional coordinates of the above-mentioned three-dimensional vector projection on a certain plane. The gaze point in this embodiment can be understood as the two-dimensional coordinates of the line of sight on the first device.

S206. Search, in the instruction database of the first device, an instruction corresponding to the attention point and the motion information as an interaction operation instruction.

In the present embodiment, the command database can be understood as a database of corresponding points of gaze points, motion information, and interoperation instructions corresponding to the eye images set in advance.

After determining the gaze point of the user on the first device, the step may search the instruction database of the first device in combination with the motion information to determine the corresponding instruction as the interoperation instruction. It can be understood that the instruction database of different devices may be different. Correspondingly, the interaction instructions determined by the fixation point and the motion information in different devices may be different.

S207. Acquire image information of the second device when the user looks at the second device.

In this embodiment, when the user looks at the second device, the step may acquire image information of the second device to determine an identity of the second device based on the image information.

S208. Search for the identifier information corresponding to the image information of the second device in the preset device database as the identity identifier of the second device.

After obtaining the image information of the second device, the step may be searched in the preset device database, and the identifier information corresponding to the image information of the second device is determined as the identity identifier of the second device, so that the identifier can be implemented based on the identifier. The interaction between the first device and the second device.

S209. Select the first device and/or the second device as the target device according to the operation corresponding to the interaction operation instruction.

In this embodiment, after the interaction operation instruction is determined, the step may determine the operation corresponding to the interaction operation instruction, and then select the first device and/or the second device as the target device for data transmission. For example, when the interworking instruction is a file sending instruction, the corresponding operation may be that the first device sends the file to the second device, and the step may select the first device as the target device. When the interworking instruction is to receive the file instruction, the corresponding operation may be that the second device sends the file to the first device. In this step, the second device may be selected as the target device.

S210. Send the identity identifier of the first device, the identity identifier of the second device, and the interaction operation instruction to the target device.

After the target device is determined, the step may send the identity of the first device, the identity of the second device, and the interworking instruction to the target device, so that the target device implements the interaction between the devices based on the received content.

FIG. 2 is a schematic diagram of an application scenario of a device control method according to Embodiment 2 of the present application. As shown in FIG. 2c, the user 220 uses the control device 221 to effect interaction between the first device 222 and the second device 223. It should be noted that FIG. 2 is only a schematic diagram of an application scenario in the second embodiment. The application scenario of the device control method in this embodiment is not limited herein. The control device 221 can be an AR glasses, and the first device 222 and the second device 223 can be devices such as a notebook, a mobile phone, or a palmtop computer. The first device 222 and the second device 223 are not defined here. The first device 222 may be a device for the user 220 to obtain an interoperation instruction, and the device that the user 220 first looks at may be the first device 222. The number of the second devices 223 is not limited. In addition, the control device 221 may also be built in each device (the first device 222 and the second device 223); it may also be externally attached to the first device 222 and the second device 223.

Based on the application scenario, the process of the device control method in this embodiment may be: when the user 220 looks at the first device 222, the image of the eye of the user 220, the motion information, and the first device may be acquired by the image capturing device in the control device 221. Image information of 222. Then, the identifier information corresponding to the image information of the first device 222 is searched for as the identifier of the first device 222 by using the preset device database, and the gaze point of the line of sight of the user 220 on the first device 222 is determined by the feature information in the eye image. An instruction corresponding to the fixation point and the motion information is searched for as an interactive operation instruction based on the instruction database of the first device 222. When the user 220 looks at the second device 223, the control device 221 can acquire the image information of the second device 223 through the image acquisition device, and determine the identity of the second device 223 based on the image information. The last control device 221 may select the first device 222 and/or the second device 223 as the target device based on the operation corresponding to the interoperation instruction, and send the identity of the first device 222, the identity of the second device 223, and the interoperation instruction. To the target device, to achieve interaction between devices.

A device control method provided in Embodiment 2 of the present application is to specify an identity identification determining operation, an interaction instruction determining operation, an identity determining operation of the second device, and an instruction sending operation of the first device. With the method, it is possible to determine the identity of the first device and the second device based on the image information acquired when the user looks at the first device and the image information acquired when the user looks at the second device, based on the user's eye. Determining, by the gaze point on the first device, the gaze point determined by the feature information in the image, and the instruction database of the first device, in conjunction with the motion information, to determine an interaction operation instruction, to identify the identity of the first device, and second based on the determined interaction operation instruction. The identity of the device and the interworking instructions are sent to realize the interaction between different devices. On the basis of extending the application scenario of the line-of-sight tracking technology, the user can get rid of the restraint of the hands when controlling the interaction between different devices, and can not contact. Each device implements the interaction between the two devices.

Embodiment 3

3 is a schematic structural diagram of a device control apparatus according to Embodiment 3 of the present application, where the apparatus is applicable to a situation in which multiple devices are controlled to interact by a line-of-sight tracking technology, where the apparatus may be implemented by software and/or hardware, and generally Integrated on the control device.

As shown in FIG. 3, the device control apparatus includes: an information acquisition module 31, an instruction determination module 32, an identification determination module 33, and a transmission module 34.

The information acquiring module 31 is configured to determine, according to image information of the first device, an identity identifier of the first device, and acquire an eye image and motion information of the user, when the user looks at the first device;

The instruction determining module 32 is configured to determine an interaction operation instruction to be executed according to the gaze point corresponding to the eye image and the motion information;

The identifier determining module 33 is configured to determine, according to image information of the second device, an identity identifier of the second device, when the user looks at the second device;

The sending module 34 is configured to send, according to the interaction operation instruction, the identity of the first device, the identity of the second device, and the interaction operation instruction to the first device and/or the And a second device, to cause the first device and/or the second device to execute the interworking instruction.

In this embodiment, the device control device first determines, by the information acquiring module 31, the identity of the first device according to the image information of the first device when the user looks at the first device, and acquires an image of the eye of the user. And the motion information; secondly, the instruction determining module 32 determines the interactive operation instruction to be executed according to the gaze point corresponding to the eye image and the motion information; and then, by the identifier determining module 33, when the user looks at the second device, according to The image information of the second device determines the identity of the second device; finally, the identity of the first device, the identity of the second device, and the location are determined by the sending module 34 based on the interaction operation instruction. The interworking instructions are sent to the first device and/or the second device to cause the first device and/or the second device to execute the interworking instructions.

The embodiment of the present invention provides a device control apparatus, which is capable of determining an interaction operation instruction to be executed between devices based on an eye image and motion information acquired by a user looking at the first device, to identify the identity of the first device based on the interaction operation instruction. And the second device identifier and the interoperation instruction are sent to the first device and/or the second device, so that the first device and/or the second device perform the interaction operation instruction to complete the interaction between the devices. In the process of controlling the device, the user's eye point is analyzed to determine the user's gaze point and the user's motion information is effectively implemented to control the interaction between multiple devices, and the application scenario of the line-of-sight tracking technology is extended. .

Further, the information acquiring module 31 is specifically configured to: when the user looks at the first device, acquire an eye image and motion information of the user, and acquire image information of the first device; and search and describe the data in the preset device database. The identification information corresponding to the image information of the first device is used as the identity identifier of the first device. Further, the motion information optimization includes: head motion information and/or gesture information.

Further, the instruction determining module 32 is specifically configured to: extract feature information in the eye image, the feature information includes a pupil feature; determine a line of sight corresponding to the eye image according to the feature information; A line of sight determines a gaze point of the user on the first device; an instruction corresponding to the gaze point and the motion information is searched for as an interactive operation instruction in an instruction database of the first device.

Further, the identifier determining module 33 is configured to: when the user looks at the second device, acquire image information of the second device; and search for a identifier corresponding to the image information of the second device in the preset device database. The information is used as the identity of the second device.

Further, the sending module 34 is specifically configured to: select the first device and/or the second device as the target device according to the operation corresponding to the interaction operation instruction; and identify the identity and location of the first device The identity of the second device and the interactive operation instruction are sent to the target device.

The device control device may perform the device control method provided by any embodiment of the present application, and has a function module and a beneficial effect corresponding to the execution method.

Embodiment 4

FIG. 4 is a schematic structural diagram of a control device according to Embodiment 4 of the present application. As shown in FIG. 4, the control device provided in Embodiment 4 of the present application includes: one or more processors 41 and storage devices 42; the processor 41 in the control device may be one or more, and one processing is performed in FIG. The device 41 is for example; the storage device 42 is configured to store one or more programs; the one or more programs are executed by the one or more processors 41 such that the one or more processors 41 implement the present application The device control method according to any of the embodiments.

The control device may further include an input device 43 and an output device 44.

The processor 41, the storage device 42, the input device 43, and the output device 44 in the control device may be connected by a bus or other means, as exemplified by a bus connection in FIG.

The storage device 42 in the control device is used as a computer readable storage medium for storing one or more programs, and the program may be a software program, a computer executable program, and a module, as in the first or second embodiment of the present application. Program instructions/modules corresponding to the device control method are provided (for example, the modules in the device control device shown in FIG. 3 include: an information acquisition module 31, an instruction determination module 32, an identification determination module 33, and a transmission module 34). The processor 41 executes various functional applications and data processing of the control device by executing software programs, instructions, and modules stored in the storage device 42, that is, implementing the device control method in the foregoing method embodiments.

The storage device 42 may include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application required for at least one function; the storage data area may store data created according to usage of the device, and the like. Moreover, storage device 42 can include high speed random access memory, and can also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other non-volatile solid state storage device. In some examples, storage device 42 may further include memory remotely located relative to processor 41, which may be connected to the device over a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

The input device 43 can be configured to receive input digital or character information, and to generate key signal input or acquisition device image information, user eye image and motion information, and the like related to user settings and function control of the control device. Input device 43 may include, but is not limited to, an image capture device (such as an infrared camera that is equipped with an infrared light), an input device such as a button and/or a microphone. Output device 44 may include, but is not limited to, a display screen.

Moreover, when one or more programs included in the control device are executed by the one or more processors 41, the program performs the following operations: when the user looks at the first device, determines the image according to the image information of the first device. An identity of the first device, and acquiring an eye image and motion information of the user; determining an interaction operation instruction to be executed according to the gaze point corresponding to the eye image and the motion information; and the user gazing at the second device Determining an identity of the second device according to the image information of the second device; and determining, according to the interaction operation instruction, the identity of the first device, the identity of the second device, and the interaction An operation instruction is sent to the first device and/or the second device to cause the first device and/or the second device to execute the interworking instruction.

In addition, the embodiment of the present application further provides a computer readable storage medium, where the computer program is stored, and when the program is executed by the processor, the method is used to execute a device control method, where the method includes: when the user looks at the first device, Determining an identity of the first device according to the image information of the first device, and acquiring an eye image and motion information of the user; determining, to be performed, according to the gaze point corresponding to the eye image and the motion information The interaction instruction of the first device is determined according to the image information of the second device when the user looks at the second device; and the identity of the first device is determined based on the interaction operation instruction, The identity identifier of the second device and the interaction operation instruction are sent to the first device and/or the second device, so that the first device and/or the second device perform the interaction operation instruction.

Optionally, when the program is executed by the processor, the program may be used to implement a device control method provided by any embodiment of the present application. Through the above description of the embodiments, those skilled in the art can clearly understand that the present application can be implemented by software and necessary general hardware, and of course can also be implemented by hardware, but in many cases, the former is a better implementation. . Based on such understanding, the technical solution of the present application, which is essential or contributes to the prior art, may be embodied in the form of a software product, which may be stored in a computer readable storage medium, such as a floppy disk of a computer. , Read-Only Memory (ROM), Random Access Memory (RAM), Flash (FLASH), hard disk or optical disk, etc., including a number of instructions to make a computer device (can be a personal computer) The server, or network device, etc.) performs the methods described in various embodiments of the present application.

Note that the above are only the preferred embodiments of the present application and the technical principles applied thereto. A person skilled in the art will understand that the present application is not limited to the specific embodiments described herein, and that various changes, modifications and substitutions can be made by those skilled in the art without departing from the scope of the invention. Therefore, although the present application has been described in detail by the above embodiments, the present application is not limited to the above embodiments, and other equivalent embodiments may be included without departing from the concept of the present application. The scope is determined by the scope of the appended claims.

Industrial applicability

The solution provided by the embodiment of the present application can be applied to the field of the Sight Tracking technology. In the embodiment of the present application, when the user looks at the first device, the identity of the first device is determined according to the image information of the first device, and Obtaining an eye image and motion information of the user; determining an interaction operation instruction to be executed according to the gaze point corresponding to the eye image and the motion information; and when the user looks at the second device, according to the image of the second device Determining, by the information, the identity of the second device; sending, according to the interaction operation instruction, the identity of the first device, the identity of the second device, and the interaction operation instruction to the first device and And the second device, in the manner of causing the first device and/or the second device to execute the interactive operation instruction, in the process of controlling the device, determining by analyzing the image of the user's eye The user's gaze point is combined with the user's motion information to realize interaction control between multiple devices, and the application scenario of the gaze tracking technology is expanded.

Claims

A device control method includes:

Determining an identity of the first device according to image information of the first device, and acquiring an eye image and motion information of the user, when the user looks at the first device;

Determining an interworking instruction to be executed according to the gaze point corresponding to the eye image and the motion information;

Determining an identity of the second device according to image information of the second device when the user is watching the second device;

Sending, according to the interaction operation instruction, the identity identifier of the first device, the identity identifier of the second device, and the interaction operation instruction to the first device and/or the second device, so as to The first device and/or the second device execute the interworking instruction.
The method of claim 1, wherein the determining the identity of the first device according to the image information of the first device comprises:

Obtaining image information of the first device;

The identifier information corresponding to the image information of the first device is searched in the preset device database as the identity identifier of the first device.
The method according to claim 1, wherein the determining the interoperation instruction to be executed according to the gaze point corresponding to the eye image and the motion information comprises:

Extracting feature information in the eye image, the feature information including a pupil feature;

Determining a line of sight corresponding to the eye image according to the feature information;

Determining a gaze point of the user on the first device according to the determined line of sight;

An instruction corresponding to the fixation point and the motion information is searched for as an interactive operation instruction in an instruction database of the first device.
The method of claim 1, wherein the determining the identity of the second device according to the image information of the second device comprises:

Obtaining image information of the second device;

And identifying, by using the preset device database, identifier information corresponding to the image information of the second device as the identity identifier of the second device.
The method of claim 1, wherein the transmitting the identity of the first device, the identity of the second device, and the interworking instruction to the first based on the interworking instruction The device and/or the second device includes:

Selecting the first device and/or the second device as the target device according to the operation corresponding to the interaction operation instruction;

Sending the identity of the first device, the identity of the second device, and the interworking instruction to the target device.
The method of claim 1, wherein the motion information comprises: head motion information and/or gesture information.
A device control device comprising:

An information acquiring module, configured to determine an identity of the first device according to image information of the first device, and obtain an eye image and motion information of the user, when the user looks at the first device;

The instruction determining module is configured to determine an interaction operation instruction to be executed according to the gaze point corresponding to the eye image and the motion information;

An identifier determining module, configured to determine, according to image information of the second device, an identity identifier of the second device when the user looks at the second device;

a sending module, configured to send, according to the interaction operation instruction, an identity of the first device, an identity of the second device, and the interaction operation instruction to the first device and/or the second a device to cause the first device and/or the second device to execute the interworking instruction.
The device according to claim 7, wherein the information acquisition module is specifically configured to:

Obtaining image information of the first device;

The identifier information corresponding to the image information of the first device is searched in the preset device database as the identity identifier of the first device.
A control device comprising:

One or more processors;

a storage device configured to store one or more programs;

The one or more programs are executed by the one or more processors, such that the one or more processors implement the device control method of any of claims 1-6.
A computer readable storage medium having stored thereon a computer program, the program being executed by a processor to implement the device control method according to any one of claims 1-6.