CN114442811A

CN114442811A - Control method and device

Info

Publication number: CN114442811A
Application number: CN202210112269.3A
Authority: CN
Inventors: 丁博; 许洪领
Original assignee: Lenovo Beijing Ltd
Current assignee: Lenovo Beijing Ltd
Priority date: 2022-01-29
Filing date: 2022-01-29
Publication date: 2022-05-06

Abstract

The present disclosure relates to a control method and a control device, the control method including: during the process that the electronic equipment executes a control function, obtaining auxiliary information, wherein the auxiliary information is different from input information for realizing the control function; and processing the input information at least based on the auxiliary information so as to enable the electronic equipment to execute control operation corresponding to the processing result.

Description

Control method and device

Technical Field

The disclosure relates to the technical field of information processing, in particular to a control method and a control device.

Background

The eye movement belongs to one of daily behaviors, the existing eye tracking technology requires a user to use a specific eye movement deliberately for a long time, the user is easy to have fatigue, and the user is easy to generate misjudgment because the user can perform some unnecessary eye movements; meanwhile, the existing eyeball tracking technology cannot distinguish unconsciousness and conscious behaviors of eyes, so that misjudgment is easily caused, and the use of the eyeball tracking technology is limited.

Disclosure of Invention

According to one aspect of the present disclosure, there is provided a control method including:

during the process that the electronic equipment executes a control function, obtaining auxiliary information, wherein the auxiliary information is different from input information for realizing the control function;

and processing the input information at least based on the auxiliary information so as to enable the electronic equipment to execute control operation corresponding to the processing result.

In some embodiments, the method further comprises:

in response to obtaining a trigger operation for enabling the electronic equipment to provide a control function, controlling the electronic equipment to execute the control function corresponding to the trigger operation, wherein the control functions corresponding to different trigger operations are different;

accordingly, the obtaining the auxiliary information includes:

and obtaining the auxiliary information at least based on the category information of the control functions, wherein the auxiliary information corresponding to different categories of control functions is different.

In some embodiments, the obtaining the assistance information based at least on the category information of the control function comprises:

determining a first component capable of acquiring input information and the auxiliary information that implement the control function based on category information of the control function, the first component belonging to or not belonging to the electronic device, controlling the first component to acquire the input information and the auxiliary information; or the like, or, alternatively,

determining a second component and a third component based on the category information of the control function, wherein the second component is used for acquiring input information for realizing the control function, the third component is used for acquiring the auxiliary information, and the second component and/or the third component belong to or do not belong to the electronic equipment;

controlling the second means to acquire the input information, and controlling the third means to acquire the auxiliary information.

In some embodiments, the obtaining the auxiliary information includes:

obtaining the auxiliary information during or after obtaining the input information; or the like, or, alternatively,

responding to the input information, wherein the corresponding control operation cannot be realized, and obtaining auxiliary information corresponding to the input information; or the like, or, alternatively,

and in response to that the variation information of the input information does not meet a preset condition, obtaining auxiliary information corresponding to the input information.

In some embodiments, processing the input information based on at least the auxiliary information to cause the electronic device to perform a control operation corresponding to a processing result includes:

determining an association relationship between the input information and an input person intention based on the auxiliary information so as to enable the electronic equipment to execute corresponding control operation based on the association relationship; or the like, or, alternatively,

editing the input information based on the auxiliary information, so that the electronic equipment executes a control operation or does not execute any operation based on the edited input information.

In some embodiments, performing a corresponding control operation based on the association includes:

if the incidence relation represents that the input information accords with the intention of an inputter, controlling the electronic equipment to execute control operation corresponding to the input information; and/or the presence of a gas in the atmosphere,

and if the incidence relation represents that the input information does not accord with the intention of the input person, controlling the electronic equipment to output prompt information.

In some embodiments, performing a control operation or not performing any operation based on the edited input information includes:

performing information enhancement on the input information, and executing corresponding control operation based on the enhanced input information; or the like, or, alternatively,

and weakening the input information, and not performing any operation based on the weakened input information.

In some embodiments, the processing the input information based on at least the auxiliary information to cause the electronic device to perform a control operation corresponding to a processing result includes:

acquiring identity information of an inputter, and processing the input information by using corresponding processing strategies based on the identity information and the auxiliary information, wherein the processing strategies corresponding to different identity information are different; or the like, or, alternatively,

obtaining historical input information of an inputter, and adjusting the processing result based on the historical input information to execute corresponding control operation based on the adjusted processing result; or the like, or, alternatively,

and obtaining configuration information of the controlled object, and adjusting the processing result based on the configuration information so as to execute corresponding control operation based on the adjusted processing result.

In some embodiments, the control function includes any one of gaze tracking control, gesture control, lip language control, voice control, or motion control;

the auxiliary information includes at least one of information characterizing attention, information characterizing sleep state, and information characterizing input parameters.

According to one of the aspects of the present disclosure, there is also provided a control apparatus including:

the acquisition module is configured to acquire auxiliary information in the process of executing a control function by the electronic equipment, wherein the auxiliary information is different from input information for realizing the control function;

the processing module is configured to process the input information at least based on the auxiliary information so as to enable the electronic equipment to execute control operation corresponding to a processing result.

An embodiment of the present disclosure further provides an electronic device, including: the control method comprises a processor and a memory, wherein the memory is used for storing computer executable instructions, and the processor realizes the control method when executing the computer executable instructions.

The embodiment of the present disclosure also provides a computer-readable storage medium, on which computer-executable instructions are stored, and when the computer-executable instructions are executed by a processor, the control method is implemented.

According to the control method, the control device, the electronic equipment and the storage medium provided by various embodiments of the disclosure, in the process that the electronic equipment executes a control function, auxiliary information corresponding to input information different from that used for realizing the control function is obtained, and the input information is processed at least based on the auxiliary information, so that more accurate input information can be obtained, and thus, the corresponding control function is accurately realized, the control accuracy of the electronic equipment is improved, and the reliability of realizing the control function is improved.

Drawings

FIG. 1 illustrates a flow chart of a control method of an embodiment of the present disclosure;

FIG. 2 illustrates a flow chart of another control method of an embodiment of the present disclosure;

fig. 3 shows a flowchart of obtaining auxiliary information in the control method of the embodiment of the present disclosure;

fig. 4 illustrates another flowchart of obtaining auxiliary information in the control method of the embodiment of the present disclosure;

fig. 5 shows a flowchart of processing input information based on auxiliary information in the control method of the embodiment of the present disclosure;

fig. 6 illustrates another flowchart of processing input information based on auxiliary information in the control method of the embodiment of the present disclosure;

fig. 7 shows a schematic structural diagram of a control device according to an embodiment of the present disclosure.

Detailed Description

Various aspects and features of the disclosure are described herein with reference to the drawings.

It will be understood that various modifications may be made to the embodiments of the present application. Accordingly, the foregoing description should not be construed as limiting, but merely as exemplifications of embodiments. Other modifications within the scope and spirit of the present disclosure will occur to those skilled in the art.

The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the disclosure and, together with a general description of the disclosure given above, and the detailed description of the embodiments given below, serve to explain the principles of the disclosure.

These and other characteristics of the present disclosure will become apparent from the following description of preferred forms of embodiment, given as non-limiting examples, with reference to the attached drawings.

It is also to be understood that although the present disclosure has been described with reference to certain specific examples, those skilled in the art will be able to ascertain many other equivalents to the present disclosure.

The above and other aspects, features and advantages of the present disclosure will become more apparent in view of the following detailed description when taken in conjunction with the accompanying drawings.

Specific embodiments of the present disclosure are described hereinafter with reference to the accompanying drawings; however, it is to be understood that the disclosed embodiments are merely exemplary of the disclosure that may be embodied in various forms. Well-known and/or repeated functions and structures have not been described in detail so as not to obscure the present disclosure with unnecessary or unnecessary detail. Therefore, specific structural and functional details disclosed herein are not to be interpreted as limiting, but merely as a basis for the claims and as a representative basis for teaching one skilled in the art to variously employ the present disclosure in virtually any appropriately detailed structure.

Fig. 1 shows a flowchart of a control method of an embodiment of the present disclosure. As shown in fig. 1, an embodiment of the present disclosure provides a control method, including the following steps:

s101: in the process of executing a control function by the electronic equipment, auxiliary information is obtained, wherein the auxiliary information is different from input information for realizing the control function.

The electronic device may include various handheld devices (e.g., mobile phones, tablet computers, PDAs, etc.) having a wireless communication function, wearable devices (e.g., smart glasses, smart watches, smart bands, pedometers, etc.), vehicle-mounted devices, computing devices, displays, all-in-one machines, and other terminal devices.

The control function includes any one of line-of-sight tracking control, attitude control, lip language control, voice control, or motion control. The electronic device can track, identify or monitor the target object through the control function. The sight tracking control function can track the eye movement of the target object in real time; the gesture control function can recognize gesture actions and the like of the target object; the lip language control function can identify the lip language of the target object; the voice control function may obtain the voice information of the target object in real time, and then control the electronic device or another electronic device to execute a corresponding function according to the voice information, for example, the electronic device may control the audio being played to adjust the volume, pause or accelerate the playing, and the like according to the voice information of the target object. The operation control function can identify the motion actions of running, jumping, fast walking and the like of the target object.

The target object may be a user using the electronic device, that is, an input person of the electronic device, for example, the electronic device may be smart glasses, the target object is a wearing user wearing the smart glasses, and the smart glasses may track an eye movement of the wearing user. For another example, the electronic device may be an intelligent bracelet, the target object is a wearing user wearing the intelligent glasses, and the intelligent bracelet may monitor the motion information and/or heartbeat and other cardiac information of the wearing user, so that the target object is convenient to perform health management.

The input information may be information such as an operation behavior and an input parameter for implementing the control function, and after receiving the input information, the electronic device converts the input information into a control instruction for executing the control function, so that the electronic device can be controlled to execute the corresponding control function according to the input information, and the control function is implemented. The input information may also be the control instruction directly, for example, after the electronic device receives a gaze tracking instruction input by an input person, the electronic device may execute a gaze tracking function to track the eye movement of the input person; in the process of tracking the sight line, the electronic device receives the tracking parameter (such as the eye movement information) of the sight line tracking input by the input person, and can track the sight line of the input person according to the eye movement information, so that the sight line tracking function is realized. The input information may also be other related information, for example, the tracking parameter may be an ambient light parameter, and the electronic device may generate a parameter adjustment instruction when receiving the ambient light parameter, and control the electronic device to adjust a parameter of a corresponding component, for example, control a gaze tracking device in the smart glasses to adjust a shading parameter, reduce interference of ambient light on the gaze tracking device, and reliably track the gaze of the input person.

The input information may be actively input based on the conscious behavior of the input person after the input person determines that the corresponding control function needs to be executed, or may be input based on the unconscious behavior of the input person, for example, when the input person mistakenly inputs the input information into the electronic device, which causes the gaze tracking control function to be activated and executed, since the electronic device does not need to execute the gaze tracking control function, the execution of the control function is invalid, and at this time, the electronic device may acquire auxiliary information corresponding to the gaze tracking control function to further determine whether the gaze tracking control function needs to be implemented, so as to prevent the control function from being triggered by mistake.

In addition, the input information may include other irrelevant information, so that the electronic device cannot accurately execute the corresponding control function, and in order to ensure that the electronic device accurately implements the corresponding control function, in this step, when the electronic device executes the control function through the input information, the auxiliary information corresponding to the control function is obtained.

Further, the auxiliary information includes at least one of information characterizing attention, information characterizing sleep state, and information characterizing input parameters.

When the auxiliary information is information representing attention, whether the input person uses the corresponding control function can be judged through the auxiliary information, for example, when the input person does not concentrate on using the sight tracking function, the electronic equipment does not execute the sight tracking function even if the input information contains eye movement information of the input person, namely, the input information is not converted into a control command for controlling the electronic equipment to execute the sight tracking function; and when the inputter is absorbed in and uses the sight tracking function, electronic equipment carries out the sight tracking function and carries out the sight tracking, so, can guarantee the accurate realization of control function, need not to track each action.

When the auxiliary information is information representing a sleep state, whether the corresponding control function is used by the input person can be judged through the auxiliary information, if the input person is in the sleep state, the input person does not use the sight line tracking function, and even if the input information includes eye information of the input person (for example, blink information, the input person may blink unconsciously in the sleep state), the electronic device does not execute the sight line tracking function. The information representing the sleep state can be brain waves, eye movements, myoelectric signals, respiration, electrocardiosignals and other data of the input person, and the sleep state of the input person can be accurately judged.

When the auxiliary information is information representing the input parameter, whether the corresponding control function is used by the inputter can be judged through the auxiliary information. For example, when the electronic device implements the gaze tracking function, it is necessary to collect not only the eye information of the input person but also the head information of the input person, that is, the information of the input parameters may include the head information of the input person.

The auxiliary information may also be other information, and the disclosure is not particularly limited. In this embodiment, the auxiliary information corresponding to different control functions is different. That is, the auxiliary information required to be obtained when different control functions are executed is different, so as to ensure the smooth implementation of the different control functions. For example, when the control function is a gaze tracking control function, the information representing attention and the information representing input parameters may be information related to eye movements, such as blinking, gaze, or eye rotation movements, and parameter information for performing corresponding movements, such as blinking frequency and gaze duration, the auxiliary information may also be information related to head movements, and the information representing sleep states may be eye movement information; when the control function is a gesture control function, the auxiliary information may be information representing attention, representing sleep state, representing input parameters and the like related to gesture actions; when the control function is a lip language control function, the auxiliary information may be information related to lip movements, for example, lip movement frequency, and emotions such as smoothness and excitement of a human body may be determined according to the lip movement frequency; when the control function is a voice control function, the auxiliary information may be information related to voice, such as volume, speed, presence or absence of a murmur, etc.; when the control function is a motion control function, the auxiliary information may be information related to the motion of the limb, for example, the motion width.

S102: and processing the input information at least based on the auxiliary information so as to enable the electronic equipment to execute the control operation corresponding to the processing result.

Specifically, after the auxiliary information is obtained, the auxiliary information can be used to perform auxiliary judgment, correction, information fusion and the like on the input information, so as to provide more accurate input information.

The processing result may be the determination result, the corrected input information, or the new input information obtained by information fusion, or may be a control command generated based on the determination result, the corrected input information, or the new input information obtained by information fusion. And the electronic equipment executes corresponding control operation according to the processing result, so that the control function is accurately realized.

According to the control method provided by the embodiment of the disclosure, in the process that the electronic device executes a control function, auxiliary information corresponding to input information different from the input information for realizing the control function is obtained, and the input information is processed at least based on the auxiliary information, so that more accurate input information can be obtained, and thus, the corresponding control function is accurately realized, the control accuracy of the electronic device is improved, and the reliability of realizing the control function is improved. In addition, the embodiment processes the input information by using the auxiliary information, and can endow the control function with more complex control logic, so that the electronic device can have richer resources to define the control function, thereby facilitating the subsequent utilization of the control function. For example, the eye tracking function can realize accurate tracking of complex eye movements, and further control the electronic device to execute other more complex control functions.

Fig. 2 shows a flow chart of another control method of an embodiment of the present disclosure. As shown in fig. 2, in some embodiments, the control method further comprises:

s103: in response to obtaining a trigger operation for enabling the electronic equipment to provide a control function, controlling the electronic equipment to execute the control function corresponding to the trigger operation, wherein the control functions corresponding to different trigger operations are different;

accordingly, in step S101, the obtaining auxiliary information includes:

s1011: and obtaining the auxiliary information at least based on the category information of the control functions, wherein the auxiliary information corresponding to different categories of control functions is different.

The triggering operation is an operation for activating the control function, and the electronic device configures corresponding triggering operations for different control functions.

The triggering operation may specifically be an operation of opening a corresponding application program (APP), the application program corresponding to the control function may be installed in advance on the electronic device, the corresponding control function is activated by an input person through the operation of opening the corresponding application program, and the electronic device may be controlled to execute the corresponding control operation under the control function by the input person inputting control information or touching a corresponding key on a page of the application program. For example, the electronic device may be pre-installed with an APP having a gaze tracking control function, an input person may activate the gaze tracking control function by opening the APP, the input person may perform various operations, such as focusing or distance detection, on the APP, and after receiving input information corresponding to the operations, the electronic device may be controlled to perform the corresponding focusing or distance detection functions. The electronic device can also be pre-installed with corresponding APPs such as a gesture control function and a lip language control function.

The triggering operation may also be an operation on a specific physical key or control, for example, a control key for implementing each control function may be preset in the electronic device, and an inputter may press the corresponding control key to activate the corresponding control function. The triggering operation may also be an operation of automatically activating the control function for different voice input operations, and the input person may directly activate the corresponding control function through the voice input, for example, the input person sends a voice command of "gaze tracking" to the electronic device, and the gaze tracking control function of the electronic device may be triggered. The triggering operation may also be other operations that activate the control function, and the disclosure is not particularly limited.

And after the corresponding control function is activated through different trigger operations, corresponding auxiliary information is obtained according to the category information of the control function. The category information of the control function is information for distinguishing different categories of control functions such as the gaze tracking control function, the gesture control function, the lip language control function, and the like.

In some embodiments, as shown in fig. 3, in step S1011, the obtaining the auxiliary information based on at least the category information of the control function includes:

s201: determining a first component capable of acquiring input information and the auxiliary information that realize the control function based on category information of the control function, the first component belonging to or not belonging to the electronic apparatus, and controlling the first component to acquire the input information and the auxiliary information.

The first component is a component for collecting the input information and the auxiliary information, and can collect both the input information for realizing the control function and the auxiliary information corresponding to the control function.

When the category of the control function is determined, a first component for obtaining input information and auxiliary information corresponding to the control function may be determined based on category information of the control function. For example, when the control function is a gaze tracking control function, the first component may be a component such as a camera or a sensor for acquiring information related to eye movement; when the control function is a voice control function, the first component may be a component for collecting voice information, such as a microphone or a sound pickup.

Different auxiliary information can be acquired through different sensors, for example, head angle change information of an inputter can be acquired through a gyroscope, head angle change speed information of the inputter is acquired through an acceleration sensor, and the head angle change information and the head angle change speed information can be used as auxiliary information representing information of input parameters to judge whether the inputter uses a sight tracking control function. When the frequency of the change of the head angle of the inputter within the preset time is greater than the preset frequency threshold value and the change of the head angle of the inputter is greater than the preset angle threshold value, the fact that the head action of the inputter frequently changes greatly can be determined, and the inputter does not use a sight tracking control function. When the head angle change information and the head angle change speed information of the input person are obtained, the head motion track of the input person can be directly obtained through the attitude sensor, and then the head angle change information and the head angle change speed information of the input person are determined based on the head motion track, so that the number of sensors is reduced, and the cost is reduced.

For another example, the head angle change information of the input person can be collected through the gyroscope, and the attention information of the input person can be obtained by detecting the heartbeat of the input person through the radar sensor, so as to judge whether the input person is attentive to use the sight tracking control function. When the head angle of the input person is not changed (not moved for a long time), although the sight line of the input person is always opposite to the screen of the electronic equipment, the heart rate is too high or too low, the attention of the input person is determined to be not focused (the concentration is low), and the input person is determined not to be focused to use the sight line tracking control function. In a specific implementation, the eye sensors can be used for acquiring the eyelid behaviors to obtain the attention information of the input person, when the blinking frequency is too low, the input person may be in a closed eye state, and it can be determined that the input person is not attentive to use the sight tracking control function.

When the blinking frequency is too low, the input person may be in an eye-open state, the eye-open state may be a state in which the input person focuses on using the gaze tracking control function (a state in which the screen is gazed at for careful learning), or may be a state in which the input person does not focus on using the gaze tracking control function (a state in which the eyes are not in the same direction), the auxiliary information may be a gazing object of the input person (for example, a screen of the electronic device, a book, or a scene outside a window, etc.) to further determine whether the input person focuses on using the gaze tracking control function, and when the input person focuses on the screen or the book, the input person may be determined to be in a state in which the input person focuses on for careful learning and needs to perform gaze tracking; when the input person is gazed outside the window, the input person can be determined to be in a vague state without performing gaze tracking.

The sensor for acquiring the auxiliary information may be any one of a radar sensor, a laser sensor, an infrared sensor, and an ultrasonic sensor.

In this embodiment, different control functions correspond to different first components, that is, the first components are used to obtain input information and auxiliary information of a certain control function. In a specific implementation, the first component may also obtain input information and auxiliary information corresponding to a plurality of control functions. For example, the first component may be a camera, and may acquire eye information, mouth information, gesture actions, or the like, so as to obtain input information and auxiliary information of different control functions.

In this embodiment, the input information and the auxiliary information of the control function are acquired by the same component, so that the input information and the auxiliary information correspond to each other one to one, and the accuracy of subsequent analysis processing is ensured.

The first component may be a component mounted on an electronic device, for example, a camera or a sensor mounted on an electronic device such as a mobile phone or smart glasses; the first component may also be an external device in wired or wireless communication connection with the electronic device, for example, the electronic device may be smart glasses, and the first component is a camera in wireless connection with the smart glasses. The first component may or may not belong to the electronic device, which may improve the applicability of the control method of the disclosed embodiments.

In other embodiments, as shown in fig. 4, in step S1011, the obtaining the auxiliary information based on at least the category information of the control function includes:

s301: determining a second component and a third component based on the category information of the control function, wherein the second component is used for acquiring input information for realizing the control function, the third component is used for acquiring the auxiliary information, and the second component and/or the third component belong to or do not belong to the electronic equipment;

s302: controlling the second means to acquire the input information, and controlling the third means to acquire the auxiliary information.

In this embodiment, the second component collects input information for realizing the control function, the third component collects corresponding auxiliary information, and the input information and the auxiliary information are collected by different components, so that different information can be accurately distinguished.

Similar to the first component, the second component and/or the third component may be a component mounted on the electronic device, or may be an external device communicatively connected to the electronic device. For the specific description of the second component and the third component, reference may be made to the above-mentioned first component, which is not described herein again.

In some embodiments, in step S101, the obtained auxiliary information may be obtained by one of the following methods:

in a first mode, the auxiliary information is obtained during or after the process of obtaining the input information.

The auxiliary information is obtained in the process of obtaining the input information, namely the auxiliary information and the input information can be simultaneously acquired, so that the processing efficiency of the electronic equipment can be improved. In other embodiments, after obtaining the input information for implementing the control function, the corresponding auxiliary information may be obtained to ensure that the auxiliary information corresponds to the input information.

In a second mode, in response to the input information failing to realize the corresponding control operation, auxiliary information corresponding to the input information is obtained.

Specifically, when the corresponding control operation cannot be realized by the input information, the auxiliary information corresponding to the input information is obtained. If the input information can implement the corresponding control operation, the auxiliary information does not need to be obtained. The control operation which cannot be realized correspondingly comprises that the corresponding control operation cannot be triggered to be executed, or the triggering execution of the control operation has errors, and the like. For example, when the gaze tracking function needs to be executed, and the gaze tracking function cannot be activated by the input information or the tracking accuracy is low after activation, the electronic device may obtain auxiliary information corresponding to the input information, perform auxiliary determination on the input information to detect whether the input information is accurate input information corresponding to activation of the gaze tracking function, and when it is determined that the input information is not accurate input information for implementing the control function (for example, input information corresponding to another control function, absence of input information, or the like), correct or fuse the input information using the auxiliary information, thereby obtaining accurate input information for implementing the gaze tracking function. By the method, auxiliary information does not need to be acquired under any condition to process the input information, the processing and operating efficiency of the electronic equipment can be improved, and the response performance of the electronic equipment can be improved.

The third method comprises the following steps: and in response to that the change information of the input information does not meet the preset condition, obtaining auxiliary information corresponding to the input information.

Specifically, the input information may be parameter information for tracking or recognizing the movement of each part of the target object (inputter) during execution of the control function, and for example, the input information may be eye movement information, head movement information, limb movement information, or the like of the target object. When the change information such as the change frequency, the change amplitude and the like of the input information does not meet the preset conditions, so that the motion of each part of the target object cannot be accurately tracked or identified, the auxiliary information can be obtained, and the input information is processed based on the auxiliary information to obtain accurate input information for tracking or identifying the target object.

In a specific embodiment, the control function is a motion control function, the input information may be a limb motion amplitude of the target object, the electronic device may determine whether the inputter performs a highlight motion according to the limb motion amplitude of the target object within a preset time, and when the limb motion amplitude within the preset time is large, the target object may be determined to perform the highlight motion; when the limb action amplitude is small within the preset time, the target object can be determined not to perform the wonderful action. The change information of the input information may be a change of a limb action amplitude, and the change of the limb action amplitude for judging whether to perform the wonderful action is within a certain range (no matter whether to perform the wonderful action, the change of the action amplitude within the preset time is small).

However, there may be both input information with large limb movement amplitude and input information with small limb movement amplitude in the preset time, and it is impossible to accurately determine whether the target object performs a highlight movement, and at this time, it may be determined that the change information of the input information does not satisfy the preset condition, for example, the limb movement amplitude is large at the first time, the limb movement amplitude is small at the second time, the limb movement amplitude is large at the third time, and the change of the limb movement amplitude is large in the preset time, and does not satisfy the preset condition. Furthermore, auxiliary information corresponding to the input information can be obtained, the auxiliary information can be duration time of the action with larger limb action amplitude, whether the human body carries out the wonderful action or not can be accurately judged according to the duration time, the duration time is longer, and the target object is determined to carry out the wonderful action.

The input information may also be control information for triggering activation of a corresponding control function, and in this case, the input information may still be eye motion information, head motion information, limb motion information, and the like of the target object.

In another embodiment, the control function is a gaze tracking control function, the input information is head motion information (e.g., head angle) of the target object, the change information of the input information is head motion change information (e.g., head angle change), when the head angle change is smaller than a preset angle threshold, it can be determined that the target object is in a gazing state, for example, the gaze is continuously fixed on the screen of the electronic device, however, it cannot be determined whether the target object is in a state of focusing on using the gaze tracking control function only through the head angle change information, and the target object may be in a vague state without performing gaze tracking. That is, when the head angle changes slightly, it is determined that the change information of the input information, the head angle, does not satisfy the preset condition, and the auxiliary information is acquired to determine whether or not specific gaze tracking is required (to implement the gaze tracking control function).

In some embodiments, as shown in fig. 5, in step S102, processing the input information based on at least the auxiliary information to enable the electronic device to perform a control operation corresponding to a processing result includes:

s1021: determining an association relationship between the input information and an input person intention based on the auxiliary information to cause the electronic device to perform a corresponding control operation based on the association relationship.

The association relationship between the input information and the intention of the input person refers to whether the operation behavior corresponding to the input information is conscious behavior or unconscious behavior performed by the input person. The electronic device may be controlled to perform a control operation corresponding to the control function if the input information is an intentional behavior of the input information inputter, and may be controlled not to perform a control operation corresponding to the control function if the input information is not an intentional behavior of the input information inputter. For example, when the electronic device performs the control function of gaze tracking, it may be determined that the gaze tracking function is not required to be implemented when it is determined that the eye opening or closing behavior of the input person is the unintentional behavior of the input person and is not the movement intentionally performed by the input person based on the auxiliary information, and at this time, the electronic device may be controlled not to perform the control operation corresponding to the gaze tracking function, and it is not required to perform the corresponding gaze tracking function based on each eye opening or eye closing movement, and thus the accuracy of gaze tracking and the execution efficiency of the electronic device may be improved.

Further, in step S1021, executing a corresponding control operation based on the association relationship, including:

Specifically, if the association represents that the input information is an conscious behavior performed by an inputter, controlling the electronic device to execute a control operation corresponding to the input information to realize a corresponding control function; if the association relation is not the conscious behavior performed by the input person, the electronic device is controlled not to generate the corresponding control instruction, that is, the control function is not executed, at this time, the electronic device may be controlled to output prompt information, the input information corresponding to the control function of the input person is prompted as unconscious operation of the input person, the control function corresponding to the input information is triggered by mistake, and the like, so that the input person closes the execution of the control function or further determines whether the control function needs to be executed.

In other embodiments, as shown in fig. 6, in step S102, processing the input information based on at least the auxiliary information, so that the electronic device executes a control operation corresponding to a processing result, includes:

s1022: editing the input information based on the auxiliary information, so that the electronic equipment executes a control operation or does not execute any operation based on the edited input information.

The editing of the input information based on the auxiliary information means performing enhancement processing or weakening processing on the input information based on the auxiliary information to improve the accuracy of the input information for realizing the control function.

Further, in step S1022, a control operation or no operation is performed based on the edited input information, including:

and carrying out information weakening on the input information, and not executing any operation based on the weakened input information.

Specifically, if the input information is close to a control instruction for implementing a certain control function in the electronic device, it is determined that the input information may be input information corresponding to implementing the control function, the input information is enhanced by using auxiliary information, and the input information is compensated to accurately correspond to the control function, so that the compensated input information can accurately execute a control operation corresponding to the control function, and further accurately implement the control function; if the input information is not matched with the control instructions corresponding to all the control functions in the electronic equipment, it is determined that the control function corresponding to the input information does not exist in the electronic equipment, and the input information may be invalid information, but the invalid information may interfere with the implementation of the control function, for example, the input information may have a certain association relationship with the input information corresponding to another control function, and the combination of the input information and the input information may cause the other control functions to be triggered by mistake. After the input information is weakened, the electronic equipment can be controlled to output prompt information, an inputter is prompted that the input information is invalid information, a control function corresponding to the input information is triggered by mistake, and the like.

In the embodiment of the disclosure, control operations such as unconscious behaviors of an inputter can be automatically removed, the inputter does not need to keep certain attention all the time to realize a corresponding control function, the use pressure of the inputter can be reduced, and the use experience is improved.

In some embodiments, in step S102, the processing the input information based on at least the auxiliary information to enable the electronic device to perform a control operation corresponding to a processing result may be implemented by one of the following manners:

in the first mode, the identity information of the input person is obtained, the input information is processed by using corresponding processing strategies based on the identity information and the auxiliary information, and the processing strategies corresponding to different identity information are different.

Specifically, after the auxiliary information corresponding to the input information is obtained, the identity information of the input person can be obtained at the same time, a more reasonable processing strategy is configured for the input person to process the input information, and the personalized use requirement of the input person is met. The identity information of the inputter may be information uniquely identifying the identity of the inputter currently using the electronic device, which is recognized through facial recognition, fingerprint recognition, or the like. For example, the focal regions of the eyeballs of different input users are different, so that when the sight line of the input user is tracked, the input information can be processed according to the auxiliary information and the focal region information of the eyeballs of the input users, and the sight line of the input user can be accurately acquired when the sight line is tracked. For another example, the eyes of some input persons may have astigmatism, so that when the gaze of the input person is tracked, the input information can be processed according to the auxiliary information and the astigmatism information of the input person, so as to avoid that the light is too dazzling and causes harm to the human body when the gaze is tracked.

The identity information of the inputter can also be information used for representing the identity of the inputter, such as the age, the height and the like of the inputter. For example, when the input person is a child, the operation behavior corresponding to the input information is an invalid operation regardless of the type of the auxiliary information, and at this time, the electronic device is directly controlled not to perform any operation without processing the input information.

In the second mode, historical input information of an input person is obtained, and the processing result is adjusted based on the historical input information so as to execute corresponding control operation based on the adjusted processing result.

The historical input information may be a historical error rate at which a control function is triggered by mistake when the input person performs a historical control operation. For example, when the input person executes the sight line tracking function based on certain input information, the probability that the control operation corresponding to the input information is erroneously triggered is high, and therefore, after the input information is processed based on the auxiliary information to obtain a processing result, the input person may output the prompt information without executing any operation so that the input person further determines whether the control operation needs to be executed.

The historical input information may also be a habit operation of the input person, for example, the habit of the input person performs gaze tracking according to a distance between an eyeball and a screen of the electronic device, so that after the input information is processed based on the auxiliary information to obtain a processing result, when performing gaze tracking control operation, the gaze tracking can be continued based on the distance between the eyeball of the input person and the screen of the electronic device, thereby making the control operation on the electronic device more suitable for the habit of the input person.

Through the history input information electronic equipment, the control functions of the electronic equipment can be accurately realized, the electronic equipment can respond to the processing result in time, and the response efficiency and reliability of the electronic equipment are improved.

And thirdly, obtaining the configuration information of the controlled object, and adjusting the processing result based on the configuration information so as to execute the corresponding control operation based on the adjusted processing result.

The controlled object is the electronic device, and the configuration information may include content configuration information, layout configuration information, control configuration information, and the like of the electronic device. For example, after the input information is processed based on the auxiliary information to obtain the processing result, the layout of the controls of the electronic device can be adjusted, and the controls which are frequently used by the input person and are not easily touched by mistake and correspond to the control function are arranged at the more prominent positions of the screen, so that the input person can conveniently operate the controls.

In the above embodiment, the input person and the target object are the same user. In a specific implementation, the input person and the target object may be different users, the input person is a user of the electronic device, the target object may be an independent user, and the target object may also be an object such as a robot. For example, the input person may control the motion of the robot through the electronic device, the auxiliary information is auxiliary information corresponding to input information for enabling the robot to perform a preset motion (for example, a pick-and-place motion), and the auxiliary information may specifically be information representing an input parameter. The control method when the input person and the target object are different may refer to the control method when the input person and the target object are the same user, and details are not described here.

Fig. 7 shows a schematic structural diagram of a control device according to an embodiment of the present disclosure. As shown in fig. 7, an embodiment of the present disclosure further provides a control device, including:

an obtaining module 10 configured to obtain auxiliary information during execution of a control function by an electronic device, wherein the auxiliary information is different from input information for realizing the control function;

and the processing module 20 is configured to process the input information at least based on the auxiliary information so as to enable the electronic equipment to execute a control operation corresponding to a processing result.

In some embodiments, the control device further comprises:

the control module is configured to respond to the trigger operation for enabling the electronic equipment to provide a control function, control the electronic equipment to execute the control function corresponding to the trigger operation, and control functions corresponding to different trigger operations are different;

accordingly, the obtaining module 10 is configured to:

In some embodiments, the obtaining module 10 is further configured to:

determining a first component capable of acquiring input information and the auxiliary information that implement the control function based on category information of the control function, the first component belonging to or not belonging to the electronic device, controlling the first component to acquire the input information and the auxiliary information; or the like, or a combination thereof,

In some embodiments, the obtaining module 10 is further configured to:

responding to the input information, wherein the corresponding control operation cannot be realized, and obtaining auxiliary information corresponding to the input information; or the like, or a combination thereof,

and in response to that the change information of the input information does not meet the preset condition, obtaining auxiliary information corresponding to the input information.

In some embodiments, the processing module 20 is specifically configured to:

In some embodiments, the processing module 20 is further configured to:

The control device provided in the embodiment of the present disclosure corresponds to the control method in the embodiment described above, and based on the control method described above, a person skilled in the art can understand the specific implementation manner of the control device in the embodiment of the present disclosure and various variations thereof, and any optional items in the embodiment of the control method are also applicable to the control device, and are not described herein again.

The processor may be a general-purpose processor, including a central processing unit CPU, a Network Processor (NP), and the like; but also a digital signal processor DSP, an application specific integrated circuit ASIC, a field programmable gate array FPGA or other programmable logic device, discrete gate or transistor logic, discrete hardware components.

The memory may include Random Access Memory (RAM) and may also include non-volatile memory (non-volatile memory), such as at least one disk memory.

The above embodiments are merely exemplary embodiments of the present disclosure, which is not intended to limit the present disclosure, and the scope of the present disclosure is defined by the claims. Various modifications and equivalents of the disclosure may occur to those skilled in the art within the spirit and scope of the disclosure, and such modifications and equivalents are considered to be within the scope of the disclosure.

Claims

1. A control method, comprising:

and processing the input information at least based on the auxiliary information so as to enable the electronic equipment to execute the control operation corresponding to the processing result.

2. The method of claim 1, further comprising:

accordingly, the obtaining the auxiliary information includes:

3. The method of claim 2, wherein the obtaining the assistance information based at least on class information of the control function comprises:

4. The method of claim 1, wherein the obtaining the assistance information comprises:

5. The method according to any one of claims 1 to 4, wherein processing the input information based on at least the auxiliary information to cause the electronic device to perform a control operation corresponding to a processing result comprises:

determining an association relationship between the input information and an input person intention based on the auxiliary information so as to enable the electronic equipment to execute corresponding control operation based on the association relationship; or the like, or a combination thereof,

6. The method of claim 5, wherein performing a corresponding control operation based on the association comprises:

7. The method of claim 5, wherein performing a control operation or not performing any operation based on the edited input information comprises:

8. The method of claim 5, wherein the processing the input information based on at least the auxiliary information to cause the electronic device to perform a control operation corresponding to a processing result comprises:

9. The method of any one of claims 1 to 4, 6 to 8, wherein the control function comprises any one of gaze tracking control, gesture control, lip language control, voice control, or motion control;

10. A control device, comprising: