WO2022193989A1 - Operation method and apparatus for electronic device and electronic device - Google Patents

Abstract

Embodiments of the present application provide an operation method and apparatus for an electronic device and the electronic device. The operation method for the electronic device comprises: the electronic device obtains pupil area information of a user using the electronic device, then determines a display position of a target key icon in a current display interface of the electronic device according to the pupil area information of the user, generates the target key icon, displays the target key icon at the display position, then measures a watching duration of the user on the target key icon, determines an operation weight for the target key icon according to the watching duration, determines an operation instruction corresponding to the operation weight, and executes the operation instruction, such that the target key icon can be watched by human eyes to operate the electronic device, reduce the use frequency of fingers when the electronic device is operated, and reduce the requirement for the quietness of an environment when the electronic device is controlled by sound.

Description

Operating method, device and electronic device for electronic equipment

This application claims the priority of the Chinese patent application with the application number 202110280578.7 and the invention titled "Operation Method, Apparatus and Electronic Equipment for Electronic Equipment" filed with the China Patent Office on March 16, 2021, the entire contents of which are incorporated by reference in in this application.

technical field

The embodiments of the present application relate to the field of communication technologies, and in particular, to an operation method, apparatus, and electronic device of an electronic device.

Background technique

With the development of modern technology, the functions of mobile phones have gradually become more diversified and diversified on the basis of their original telephone communication, and have become an indispensable part of people's lives. At the same time, the convenience and ease of using mobile phones have gradually become a Hot spots that people chase.

Existing relatively mature mobile phone operation methods include voice control and/or finger operation (including: sliding, light pressing and/or heavy pressing). However, these two operation methods also have certain limitations. The sound control method has a high demand for the quietness of the environment, and it is almost impossible to use sound for control in a relatively noisy environment. Input and output greatly limit the operable space of sound control, resulting in the existing sound control method being far less convenient than finger operation. The finger operation method requires the use of human fingers or limbs for control. For some disabled people and/or patients with certain diseases (eg, Parkinson's disease or stroke), it is almost impossible to use fingers to operate a mobile phone. At the same time, using a finger to operate the mobile phone for a long time will cause the user to experience symptoms of finger fatigue and/or finger numbness. In addition, finger operation in a cold environment will also bring discomfort to the user.

SUMMARY OF THE INVENTION

The embodiments of the present application provide an electronic device operating method, device, and electronic device, so that a user can operate the electronic device through eye movement and gaze, reduce the frequency of use of fingers when operating the electronic device, and reduce sound control of the electronic device. The requirement for the quietness of the environment when the equipment is used.

In a first aspect, an embodiment of the present application provides an operation method for an electronic device, including: acquiring pupil area information of a user using the electronic device; Displaying the display position in the interface; generating the target button icon, and displaying the target button icon at the display position; detecting the gaze duration of the user on the target button icon; determining according to the gaze duration For the operation weight of the target button icon; determine the operation instruction corresponding to the operation weight, and execute the operation instruction.

In the operation method of the electronic device, the electronic device obtains the pupil area information of the user who uses the electronic device, and then determines the display position of the target button icon in the current display interface of the electronic device according to the pupil area information of the user, and then the electronic device Generate a target button icon, display the target button icon at the above-mentioned display position, and then detect the user's gaze duration on the above-mentioned target button icon, determine the operation weight for the above-mentioned target button icon according to the above-mentioned gaze duration, and determine the corresponding operation weight. The above operation instructions are executed, so that the human eye can watch the target button icon to realize the operation of the electronic device, reduce the frequency of use of fingers when operating the electronic device, and reduce the sound when controlling the electronic device. The demand for the quietness of the environment .

In one possible implementation manner, the determining the display position of the target button icon on the current display interface of the electronic device according to the pupil area information of the user includes: according to the pupil area information of the user, using a preset The trained machine learning model obtains the display position of the target button icon in the display interface.

In one possible implementation manner, the acquiring the pupil area information of the user using the electronic device includes: capturing the area where the pupil of the user is located by using a sensor in the electronic device; The pupil is marked in the area where the pupil is located and the pupil area information of the user is obtained.

In one possible implementation manner, the obtaining the display position of the target button icon in the display interface by using a pre-trained machine learning model according to the pupil area information of the user includes: according to the pupil area information of the user area information, calculate the distance of the user's pupil center point relative to the horizontal direction of the sensor, and the angle of the user's pupil center point relative to the vertical direction of the sensor; according to the distance and the angle, use the pre-trained The machine learning model obtains the display position of the target button icon in the display interface.

In one possible implementation manner, the determining the operation instruction corresponding to the operation weight includes: when the operation weight is at the first level, determining that the operation instruction corresponding to the operation weight is an operation corresponding to the click operation When the operation weight is the second level, it is determined that the operation instruction corresponding to the operation weight is the operation instruction corresponding to the long-press operation; when the operation weight is the first level combined with the second level, it is determined that the operation instruction corresponding to the The operation instruction corresponding to the operation weight is an operation instruction corresponding to the sliding operation.

In one possible implementation manner, after determining the operation weight for the target button icon according to the gaze duration, the method further includes: adjusting the display effect of the target button icon according to the operation weight; wherein, The display effect includes color, size and/or brightness.

In a second aspect, an embodiment of the present application provides an operating device for an electronic device, including: an acquisition module for acquiring pupil area information of a user using the electronic device; a determination module for determining according to the pupil area information of the user The display position of the target button icon in the current display interface of the electronic device; the generating module is used for generating the target button icon; the display module is used for displaying the generating module at the display position determined by the determining module the generated target button icon; a detection module for detecting the gaze duration of the user on the target button icon; the determining module is further configured to determine the operation weight for the target button icon according to the gaze duration; and determining an operation instruction corresponding to the operation weight; an execution module, configured to execute the operation instruction.

In one possible implementation manner, the determining module is specifically configured to obtain the display position of the target button icon in the display interface by using a pre-trained machine learning model according to the pupil area information of the user.

In one possible implementation manner, the acquisition module is specifically configured to capture the area where the pupil of the user is located through a sensor in the electronic device; and through a collector in the electronic device, in the area where the pupil is located The pupils are marked and the pupil area information of the user is obtained.

In one possible implementation manner, the determining module includes: a calculating sub-module, configured to calculate the distance between the pupil center point of the user and the sensor in the horizontal direction according to the pupil area information of the user, and the The angle of the center point of the pupil of the user relative to the vertical direction of the sensor; the position determination submodule is used to obtain the position of the target button icon in the display interface by using the pre-trained machine learning model according to the distance and the angle. Display location.

In one possible implementation manner, the determination module includes: an instruction determination sub-module, configured to determine that the operation instruction corresponding to the operation weight is the operation instruction corresponding to the click operation when the operation weight is at the first level When the operation weight is the second level, it is determined that the operation instruction corresponding to the operation weight is the operation instruction corresponding to the long-press operation; when the operation weight is the first level combined with the second level, it is determined The operation instruction corresponding to the operation weight is the operation instruction corresponding to the sliding operation.

In one possible implementation manner, the apparatus further includes: an adjustment module, configured to adjust the operation weight of the target button icon according to the operation weight after the determination module determines the operation weight of the target button icon. Display effect; wherein, the display effect includes color, size and/or brightness.

In a third aspect, embodiments of the present application provide an electronic device, including: at least one processor; a sensor and a collector connected in communication with the processor; and at least one memory in communication with the processor, wherein: the The memory stores program instructions executable by the processor, and the processor invokes the program instructions to execute the method provided in the first aspect.

In a fourth aspect, an embodiment of the present application provides a non-transitory computer-readable storage medium, where the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions cause the computer to execute the method provided in the first aspect.

It should be understood that the second to fourth aspects of the embodiments of the present application are consistent with the technical solutions of the first aspect of the embodiments of the present application, and the beneficial effects obtained by various aspects and corresponding feasible implementations are similar, and will not be repeated.

Description of drawings

In order to illustrate the technical solutions of the embodiments of the present application more clearly, the following briefly introduces the accompanying drawings used in the embodiments. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can also be obtained from these drawings without any creative effort.

FIG. 1 is a flowchart of an operation method of an electronic device provided by an embodiment of the present application;

2 is a schematic diagram of a target button icon provided by an embodiment of the present application;

3 is a flowchart of an operation method of an electronic device provided by another embodiment of the present application;

FIG. 4 is a flowchart of an operation method of an electronic device provided by yet another embodiment of the present application;

5 is a schematic structural diagram of an operating device of an electronic device provided by an embodiment of the present application;

6 is a schematic structural diagram of an operating device of an electronic device provided by another embodiment of the present application;

FIG. 7 is a schematic structural diagram of an electronic device according to an embodiment of the present application.

Detailed ways

In order to better understand the technical solutions of the present application, the embodiments of the present application are described in detail below with reference to the accompanying drawings.

It should be clear that the described embodiments are only a part of the embodiments of the present application, but not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present application.

The terms used in the embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to limit the present application. As used in the embodiments of this application and the appended claims, the singular forms "a," "the," and "the" are intended to include the plural forms as well, unless the context clearly dictates otherwise.

In the related art in the prior art, the operation methods of electronic devices generally include voice control and/or finger operation. As mentioned above, these two operation methods have certain limitations. Compared with the voice control and/or finger operation methods, since the user always pays attention to the electronic device when operating the electronic device, the movements of the human eye are much more subtle and easier than the fingers, and there is no requirement for the quietness of the environment. The existing solution of operating electronic devices through eye movement requires embedding a vision system micro-processing chip, a vision sensor, and a vision collector in the electronic device, obtaining the user's eye movement track information through the vision collector, and sending the movement track information. To the vision sensor, the vision sensor sends the motion trajectory information to the vision system micro-processing chip, which is processed by the vision micro-processing chip to generate information instruction data and send it to the microcontroller in the electronic device; the microcontroller then calls the vision system. The program algorithm memory compares and analyzes the above information and instruction data, obtains control instructions, and sends them to the central processing unit (CPU) of the electronic equipment; the CPU of the electronic equipment operates the mobile phone according to the control instructions.

The existing implementation scheme captures the movement trajectory of the eyeball, sends the movement trajectory as an input to the controller, and converts it into a control command and sends it to the CPU of the electronic device. There are great challenges in the accuracy and variety of operations.

The embodiment of the present application provides an operation method of an electronic device, which uses eyeballs in combination with machine learning to operate the electronic device. Specifically, a visual acquisition sensor is used to collect the center position of the pupil of the human eye as an input, and the coordinates of the user's gaze at the display interface of the electronic device are used as an output. , using the machine learning model to repeatedly train the input and output to establish the input-output mapping. Then, by detecting the movements of the user's eyeballs moving up, down, left and right and/or gazing, operations such as the user's finger sliding up and down, left and right, light pressing and/or heavy pressing are simulated.

FIG. 1 is a flowchart of an operation method of an electronic device provided by an embodiment of the present application. As shown in FIG. 1 , the operation method of the electronic device may include:

Step 101, the electronic device acquires pupil area information of a user who uses the electronic device.

In this embodiment, a sensor and a collector that can capture the pupil of a human eye are embedded in the electronic device. In this way, obtaining the pupil area information of the user using the electronic device by the electronic device may be: the electronic device captures the area where the pupil of the user is located through the sensor in the electronic device; and then, through the collector in the electronic device, in the area where the pupil is located Mark the pupil and get the pupil area information of the above user.

During specific implementation, the electronic device may use the above collector, combined with a neural network algorithm, to mark the pupil in the area where the pupil is located, and obtain the pupil area information of the above user.

The above electronic device may be an intelligent terminal device such as a smart phone, a tablet computer, a smart watch, or a vehicle-mounted device, and the specific type of the above electronic device is not limited in this embodiment.

In step 102, the electronic device determines the display position of the target button icon in the current display interface of the electronic device according to the pupil area information of the user.

Step 103, the electronic device generates the above-mentioned target button icon, and displays the target button icon at the above-mentioned display position.

Referring to FIG. 2, FIG. 2 is a schematic diagram of a target button icon provided by an embodiment of the application. As shown in FIG. 2, assuming that the user looks at the icon of application 1 in the current display interface of the electronic device, then according to the user's pupil area information , after determining the display position of the target button icon, the electronic device may generate the target button icon, and display the target button icon 21 at the above-mentioned display position. In this embodiment, the shape of the above-mentioned target button icon 21 may be circular, the diameter may be 10 mm, and the color is white and translucent; however, this embodiment is not limited to this, and the size of the target button icon may also be 5 mm Or 6 mm and other sizes, the shape can also be other shapes such as rectangle or triangle, and the color can also be red or yellow. This embodiment does not limit the shape, size and/or color of the above target button icon.

During specific implementation, in order not to block the display content in the current display interface of the electronic device, the color of the target button icon 21 generally does not select a solid color, and is in a semi-transparent state as much as possible.

Step 104, the electronic device detects the user's gaze duration on the target button icon.

Step 105, the electronic device determines the operation weight for the target button icon according to the gaze duration.

Step 106, the electronic device determines an operation instruction corresponding to the above-mentioned operation weight, and executes the above-mentioned operation instruction.

Specifically, the electronic device may determine that the operation instruction corresponding to the above-mentioned operation weight is: when the operation weight is the first level, determine that the operation instruction corresponding to the above-mentioned operation weight is the operation instruction corresponding to the click operation; when the above-mentioned operation weight is the second level When the above-mentioned operation weight is the first level combined with the second level, it is determined that the operation instruction corresponding to the above-mentioned operation weight is the operation corresponding to the sliding operation. instruction. Wherein, the above-mentioned first level may be lower or higher than the second level, which is not limited in this embodiment, and is described below by taking an example that the first level is lower than the second level.

Referring to FIG. 2, assuming that the user is looking at the icon of the application 1 in the current display interface of the electronic device, the electronic device can determine the display position of the target button icon according to the pupil area information of the user, and then display the target button icon 21 at the above-mentioned display position. . Then, the electronic device detects the gaze duration of the user on the target button icon 21, and determines the operation weight for the target button icon according to the gaze duration. In specific implementation, when determining the operation weight, the above operation weight can be divided into three levels: low, medium and high according to the gaze duration from short to long. For example, when the gaze duration is less than 2 seconds, the operation weight is determined to be a low level ; When the fixation duration is greater than or equal to 2 seconds and less than 3.6 seconds, the operation weight is determined to be a medium level; when the fixation duration is greater than or equal to 3.6 seconds, the operation weight is determined to be a high level. Among them, when the operation weight is low, the operation is invalid; when the operation weight is medium, it simulates the operation of clicking the screen with a finger; when the operation weight is high, it simulates the operation of long pressing the screen; when the operation weight is high and medium For example, the user stares at the target button icon 21 for 3.6 seconds, and then stares at the target button icon 21 for 2 seconds, simulating the operation of sliding the screen with a finger. Therefore, the human eye can watch the target button icon 21 to achieve the same effect as a finger tap, a long press and a sliding screen.

In the operation method of the electronic device, the electronic device obtains the pupil area information of the user who uses the electronic device, and then determines the display position of the target button icon in the current display interface of the electronic device according to the pupil area information of the user, and then the electronic device Generate a target button icon, display the target button icon at the above-mentioned display position, and then detect the user's gaze duration on the above-mentioned target button icon, determine the operation weight for the above-mentioned target button icon according to the above-mentioned gaze duration, and determine the corresponding operation weight. The operation instruction is executed, so that the human eye can watch the target button icon to realize the operation of the electronic device, reduce the frequency of finger use when operating the electronic device, and reduce the sound when controlling the electronic device. The need for quietness of the environment .

FIG. 3 is a flowchart of an operation method of an electronic device provided by another embodiment of the present application. As shown in FIG. 3 , in the embodiment shown in FIG. 1 of the present application, step 102 may be:

Step 301 , according to the above-mentioned pupil area information of the user, using a pre-trained machine learning model to obtain the display position of the target button icon in the above-mentioned display interface.

Specifically, according to the pupil area information of the user, using a pre-trained machine learning model to obtain the display position of the target button icon in the display interface may be: according to the pupil area information of the user, calculate the relative sensor level of the pupil center point of the user direction distance, and the angle of the user's pupil center point relative to the vertical direction of the sensor; according to the above distance and the above angle, use the pre-trained machine learning model to obtain the display position of the target button icon in the above display interface.

In specific implementation, a predetermined number (for example: 100) of participant databases can be used for training. Taking the current display interface of the electronic device as shown in Figure 2 as an example, the participants can adopt random postures and watch the current display of the electronic device from different positions. Application 1 in the interface. The electronic device is provided with a sensor and a collector that can capture the pupil of the human eye. The above sensor can be used to capture the area where the pupil is located, and then the collector can be used in conjunction with the neural network algorithm to mark the pupil and obtain the pupil area information of the above-mentioned user. Passed to the CPU of the electronics. Then, the CPU of the electronic device calculates the distance of the pupil center relative to the horizontal direction of the sensor and the angle of the pupil center relative to the vertical direction through the obtained pupil area information, and uses this as an input, and takes the position of Application 1 in the current display interface of the electronic device as output. Let the participants pay attention to the application one, two, three... . Using the data in the database, through the training of a large number of machine learning models, the mapping between the input and the output is established to obtain the machine learning model. Then, use the data in the above database to test the machine learning model obtained by training, repeat the above process, let different participants look at each area in the current display interface of the electronic device one by one, and view the output of the machine learning model obtained by training. , filter out abnormal data output, and obtain the final trained machine learning model.

In this way, in the subsequent use, after the electronic device obtains the pupil area information of the user, according to the pupil area information of the user, the distance between the pupil center point of the user and the sensor in the horizontal direction and the angle between the pupil center point of the user and the sensor in the vertical direction are calculated. , and then input the above distance and the above angle into the trained machine learning model, and the machine learning model can output the position where the pupil of the human eye looks at the current display interface of the electronic device, and this position is the target button icon in the current display interface of the electronic device. display position.

FIG. 4 is a flowchart of an operation method of an electronic device provided by another embodiment of the present application. As shown in FIG. 4 , in the embodiment shown in FIG. 1 of the present application, after step 105, the method may further include:

Step 401, the electronic device adjusts the display effect of the target button icon according to the above-mentioned operation weight; wherein, the above-mentioned display effect includes color, size and/or brightness.

Specifically, taking the first level lower than the second level as an example, assuming that the color of the target button icon is white and translucent, the brightness of the target button icon can be adjusted according to the operation weight. When the first level and the first level are combined with the second level and the second level, the brightness of the target button icon is gradually increased. To put it simply, as the duration of the user's gaze at the target button icon increases, the brightness of the target button icon gradually increases;

Alternatively, the color of the target button icon can be adjusted according to the operation weight. For example, when the operation weight is the first level, the first level combined with the second level and the second level, the color of the target button icon gradually deepens;

Alternatively, the color of the target button icon can be adjusted according to the operation weight. For example, when the operation weight is the first level, the first level combined with the second level and the second level, the target button icon gradually increases.

Of course, according to the operation weight, the color of the target button icon can be adjusted as well as the size and brightness of the target button icon. For example, as the user's gaze time increases, the color of the target button icon can be gradually deepened according to the operation weight. , but keep the translucent shape, size and color change synchronously, the target button icon can be gradually enlarged from 5mm, up to 10mm; at the same time, the brightness of the target button icon gradually increases.

In addition, in addition to color, size, and brightness, the above-mentioned display effect may also include other display effects such as animation display, and the form of the above-mentioned display effect is not limited in this embodiment.

In the operation method of the electronic device provided by the embodiment of the present application, capturing the eye movement trajectory is changed to capturing the pupil center point, and the trajectory is changed to point coordinates, thereby reducing complex operations and reducing the computational burden of the processor in the electronic device. Secondly, assign a target button icon with a weight according to the gaze duration of the pupil area to the point coordinates, and differentiate invalid press, click and long press operations by the weight, so that the eye movement can be combined with effective gaze in a certain area, simulating the operation of electronic devices with fingers Effect.

The operating method of the electronic device provided by the embodiment of the present application can solve the problem that the user cannot operate the electronic device conveniently, dexterously, and easily by using the finger and/or the voice in the following scenarios:

(1) In cold climates, when wearing gloves outdoors is inconvenient to use fingers to operate electronic devices dexterously;

(2) In cold climates, when it is inconvenient to operate two-handed electronic devices indoors;

(3) Tired and bedridden, or when it is inconvenient to operate a single-handed electronic device while lying down;

(4) When it is noisy outdoors, when taking public transportation, standing in a bus or subway car, and it is inconvenient to use both hands and/or use voice to control electronic devices;

(5) The environment is noisy, it is inconvenient to operate electronic equipment with two hands, and it is not suitable for using sound to control electronic equipment;

(6) Scenarios where (2) and (3) are combined.

In addition, the operation method of the electronic device provided by the embodiments of the present application can also be used for the following special users to solve the problem that the electronic device cannot be used and is inconvenient to operate the electronic device with fingers and/or voice:

(1) Users with stroke, Parkinson’s or ALS, etc. who cannot use their fingers to perform precise swipe, click and/or long-press operations like normal people;

(2) Users with disabilities in fingers or arms (cannot use fingers to operate electronic devices normally);

(3) users with dysphonia or deaf-mute patients (difficulty using voice to control electronic equipment), and belong to users of (1) and/or (2);

(4) Users who do not speak Mandarin and have one or a combination of (1), (2) and (3) problems.

The foregoing describes specific embodiments of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that in the embodiments and still achieve desirable results. Additionally, the processes depicted in the figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

FIG. 5 is a schematic structural diagram of an operating device of an electronic device provided by an embodiment of the present application. As shown in FIG. 5 , the operating device of the electronic device may include: an acquisition module 51, a determination module 52, a generation module 53, a display module 54, detection module 55 and execution module 56;

Wherein, the obtaining module 51 is used to obtain the pupil area information of the user who uses the electronic device;

The determination module 52 is used for determining the display position of the target button icon in the current display interface of the electronic device according to the pupil area information of the above-mentioned user;

A generating module 53 is used to generate a target button icon;

The display module 54 is used to display the target button icon generated by the generation module 53 at the display position determined by the determination module 52;

The detection module 55 is used to detect the user's gaze duration on the target button icon;

The determination module 52 is further configured to determine the operation weight for the target button icon according to the above-mentioned gaze duration; and to determine the operation instruction corresponding to the above-mentioned operation weight;

The execution module 56 is configured to execute the above operation instruction.

The operating device of the electronic device provided by the embodiment shown in FIG. 5 can be used to implement the technical solution of the method embodiment shown in FIG. 1 of the present application, and the implementation principle and technical effect can be further referred to the relevant description in the method embodiment.

6 is a schematic structural diagram of an operating device of an electronic device provided by another embodiment of the present application. In this embodiment, the determination module 52 is specifically configured to obtain a target key by using a pre-trained machine learning model according to the pupil area information of the user. The display position of the icon in the above display interface.

The acquisition module 51 is specifically configured to capture the area where the pupil of the user is located through the sensor in the electronic device; and through the collector in the electronic device, mark the pupil in the area where the pupil is located and obtain the pupil area information of the user.

In this embodiment, the determination module 52 may include: a calculation sub-module 521 and a position determination sub-module 522;

The calculation submodule 521 is used to calculate the distance of the user's pupil center point relative to the horizontal direction of the sensor and the angle of the user's pupil center point relative to the vertical direction of the sensor according to the pupil area information of the user;

The position determination sub-module 522 is configured to obtain the display position of the target button icon in the above-mentioned display interface by using the pre-trained machine learning model according to the above-mentioned distance and the above-mentioned angle.

In this embodiment, the determining module 52 may include: an instruction determining sub-module 523;

The instruction determination sub-module 523 is used to determine the operation instruction corresponding to the operation weight as the operation instruction corresponding to the click operation when the operation weight is the first level; when the operation weight is the second level, determine the operation instruction corresponding to the operation weight is the operation instruction corresponding to the long-press operation; when the operation weight is the first level combined with the second level, it is determined that the operation instruction corresponding to the operation weight is the operation instruction corresponding to the sliding operation.

Further, the operating device of the electronic device may further include: an adjustment module 57;

The adjustment module 57 is configured to adjust the display effect of the target button icon according to the operation weight after the determination module 52 determines the operation weight for the target button icon; wherein the display effect includes color, size and/or brightness.

The operating device of the electronic device provided by the embodiment shown in FIG. 6 can be used to implement the technical solutions of the method embodiments shown in FIG. 1 to FIG. 4 of the present application, and the implementation principle and technical effect can be further referred to the related descriptions in the method embodiments.

FIG. 7 is a schematic structural diagram of an electronic device provided by an embodiment of the application. As shown in FIG. 7 , the electronic device may include at least one processor; a sensor and a collector connected in communication with the processor; and communicating with the processor The connected at least one memory, wherein: the memory stores program instructions executable by the processor, and the processor invokes the program instructions to execute the operation method of the electronic device provided by the embodiments shown in FIG. 1 to FIG. 4 of the present application.

The above electronic device may be an intelligent terminal device such as a smart phone, a tablet computer, a smart watch, or a vehicle-mounted device, and the specific type of the above electronic device is not limited in this embodiment.

Exemplarily, FIG. 7 shows a schematic structural diagram of an electronic device by taking a smartphone as an example. As shown in FIG. 7 , the electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, and a universal serial bus (universal serial bus). bus, USB) interface 130, charging management module 140, power management module 141, battery 142, antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, audio module 170, speaker 170A, receiver 170B, microphone 170C, earphones Interface 170D, sensor module 180, collector 181, button 190, motor 191, indicator 192, camera 193, display screen 194, and subscriber identification module (SIM) card interface 195 and the like.

The sensor module 180 includes a sensor capable of capturing the pupil of the human eye, and the collector 181 can capture the pupil of the human eye.

It can be understood that the structures illustrated in the embodiments of the present application do not constitute a specific limitation on the electronic device 100 . In other embodiments of the present application, the electronic device 100 may include more or less components than shown, or combine some components, or separate some components, or arrange different components. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (application processor, AP), a modem processor, a graphics processor (graphics processing unit, GPU), an image signal processor (image signal processor, ISP), controller, video codec, digital signal processor (digital signal processor, DSP), baseband processor, and/or neural-network processing unit (neural-network processing unit, NPU), etc. Wherein, different processing units may be independent devices, or may be integrated in one or more processors.

The controller can generate an operation control signal according to the instruction operation code and timing signal, and complete the control of fetching and executing instructions.

A memory may also be provided in the processor 110 for storing instructions and data. In some embodiments, the memory in processor 110 is cache memory. This memory may hold instructions or data that have just been used or recycled by the processor 110 . If the processor 110 needs to use the instruction or data again, it can be called directly from the memory. Repeated accesses are avoided and the latency of the processor 110 is reduced, thereby increasing the efficiency of the system.

The processor 110 executes various functional applications and data processing by running programs stored in the internal memory 121 , for example, implementing the processing performance improvement method provided by the embodiments shown in FIGS. 1 to 4 of the present application.

In some embodiments, the processor 110 may include one or more interfaces. The interface may include an integrated circuit (inter-integrated circuit, I2C) interface, an integrated circuit built-in audio (inter-integrated circuit sound, I2S) interface, a pulse code modulation (pulse code modulation, PCM) interface, a universal asynchronous transceiver (universal asynchronous transmitter) receiver/transmitter, UART) interface, mobile industry processor interface (MIPI), general-purpose input/output (GPIO) interface, subscriber identity module (SIM) interface, and / or universal serial bus (universal serial bus, USB) interface, etc.

The charging management module 140 is used to receive charging input from the charger. The charger may be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 may receive charging input from the wired charger through the USB interface 130 . In some wireless charging embodiments, the charging management module 140 may receive wireless charging input through a wireless charging coil of the electronic device 100 . While the charging management module 140 charges the battery 142 , the electronic device 100 can also be powered by the power management module 141 .

The power management module 141 is used for connecting the battery 142 , the charging management module 140 and the processor 110 . The power management module 141 receives input from the battery 142 and/or the charging management module 140, and supplies power to the processor 110, the internal memory 121, the display screen 194, the camera 193, and the wireless communication module 160. The power management module 141 can also be used to monitor parameters such as battery capacity, battery cycle times, battery health status (leakage, impedance). In some other embodiments, the power management module 141 may also be provided in the processor 110 . In other embodiments, the power management module 141 and the charging management module 140 may also be provided in the same device.

The wireless communication function of the electronic device 100 may be implemented by the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modulation and demodulation processor, the baseband processor, and the like.

Antenna 1 and Antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 may be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve antenna utilization. For example, the antenna 1 can be multiplexed as a diversity antenna of the wireless local area network. In other embodiments, the antenna may be used in conjunction with a tuning switch.

The mobile communication module 150 may provide wireless communication solutions including 2G/3G/4G/5G etc. applied on the electronic device 100 . The mobile communication module 150 may include at least one filter, switch, power amplifier, low noise amplifier (LNA) and the like. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, filter and amplify the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and then turn it into an electromagnetic wave for radiation through the antenna 1 . In some embodiments, at least part of the functional modules of the mobile communication module 150 may be provided in the processor 110 . In some embodiments, at least part of the functional modules of the mobile communication module 150 may be provided in the same device as at least part of the modules of the processor 110 .

The modem processor may include a modulator and a demodulator. Wherein, the modulator is used to modulate the low frequency baseband signal to be sent into a medium and high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low frequency baseband signal. Then the demodulator transmits the demodulated low-frequency baseband signal to the baseband processor for processing. The low frequency baseband signal is processed by the baseband processor and passed to the application processor. The application processor outputs sound signals through audio devices (not limited to the speaker 170A, the receiver 170B, etc.), or displays images or videos through the display screen 194 . In some embodiments, the modem processor may be a stand-alone device. In other embodiments, the modulation and demodulation processor may be independent of the processor 110, and be provided in the same device as the mobile communication module 150 or other functional modules.

The wireless communication module 160 can provide applications on the electronic device 100 including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), bluetooth (BT), global navigation satellites System (global navigation satellite system, GNSS), frequency modulation (frequency modulation, FM), near field communication technology (near field communication, NFC), infrared technology (infrared, IR) and other wireless communication solutions. The wireless communication module 160 may be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2 , frequency modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 110 . The wireless communication module 160 can also receive the signal to be sent from the processor 110 , perform frequency modulation on it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2 .

In some embodiments, the antenna 1 of the electronic device 100 is coupled with the mobile communication module 150, and the antenna 2 is coupled with the wireless communication module 160, so that the electronic device 100 can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), broadband Code Division Multiple Access (WCDMA), Time Division Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), BT, GNSS, WLAN, NFC , FM, and/or IR technology, etc. The GNSS may include global positioning system (global positioning system, GPS), global navigation satellite system (global navigation satellite system, GLONASS), Beidou navigation satellite system (beidou navigation satellite system, BDS), quasi-zenith satellite system (quasi -zenith satellite system, QZSS) and/or satellite based augmentation systems (SBAS).

The electronic device 100 implements a display function through a GPU, a display screen 194, an application processor, and the like. The GPU is a microprocessor for image processing, and is connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs that execute program instructions to generate or alter display information.

Display screen 194 is used to display images, videos, and the like. Display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (active-matrix organic light). emitting diode, AMOLED), flexible light-emitting diode (flex light-emitting diode, FLED), Miniled, MicroLed, Micro-oLed, quantum dot light-emitting diode (quantum dot light emitting diodes, QLED) and so on. In some embodiments, the electronic device 100 may include one or N display screens 194 , where N is a positive integer greater than one.

The electronic device 100 may implement a shooting function through an ISP, a camera 193, a video codec, a GPU, a display screen 194, an application processor, and the like.

The ISP is used to process the data fed back by the camera 193 . For example, when taking a photo, the shutter is opened, the light is transmitted to the camera photosensitive element through the lens, the light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to the ISP for processing, and converts it into an image visible to the naked eye. ISP can also perform algorithm optimization on image noise, brightness, and skin tone. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, the ISP may be provided in the camera 193 .

Camera 193 is used to capture still images or video. The object is projected through the lens to generate an optical image onto the photosensitive element. The photosensitive element may be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then transmits the electrical signal to the ISP to convert it into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. DSP converts digital image signals into standard RGB, YUV and other formats of image signals. In some embodiments, the electronic device 100 may include 1 or N cameras 193 , where N is a positive integer greater than 1.

A digital signal processor is used to process digital signals, in addition to processing digital image signals, it can also process other digital signals. For example, when the electronic device 100 selects a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy and so on.

Video codecs are used to compress or decompress digital video. The electronic device 100 may support one or more video codecs. In this way, the electronic device 100 can play or record videos of various encoding formats, such as: Moving Picture Experts Group (moving picture experts group, MPEG) 1, MPEG2, MPEG3, MPEG4 and so on.

The NPU is a neural-network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transfer mode between neurons in the human brain, it can quickly process the input information, and can continuously learn by itself. Applications such as intelligent cognition of the electronic device 100 can be implemented through the NPU, such as image recognition, face recognition, speech recognition, text understanding, and the like.

The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100 . The external memory card communicates with the processor 110 through the external memory interface 120 to realize the data storage function. For example to save files like music, video etc in external memory card.

Internal memory 121 may be used to store computer executable program code, which includes instructions. The internal memory 121 may include a storage program area and a storage data area. The storage program area can store an operating system, an application program required for at least one function (such as a sound playback function, an image playback function, etc.), and the like. The storage data area may store data (such as audio data, phone book, etc.) created during the use of the electronic device 100 and the like. In addition, the internal memory 121 may include high-speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, universal flash storage (UFS), and the like. The processor 110 executes various functional applications and data processing of the electronic device 100 by executing instructions stored in the internal memory 121 and/or instructions stored in a memory provided in the processor.

The electronic device 100 may implement audio functions through an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, an application processor, and the like. Such as music playback, recording, etc.

The audio module 170 is used for converting digital audio information into analog audio signal output, and also for converting analog audio input into digital audio signal. Audio module 170 may also be used to encode and decode audio signals. In some embodiments, the audio module 170 may be provided in the processor 110 , or some functional modules of the audio module 170 may be provided in the processor 110 .

Speaker 170A, also referred to as a "speaker", is used to convert audio electrical signals into sound signals. The electronic device 100 can listen to music through the speaker 170A, or listen to a hands-free call.

The receiver 170B, also referred to as "earpiece", is used to convert audio electrical signals into sound signals. When the electronic device 100 answers a call or a voice message, the voice can be answered by placing the receiver 170B close to the human ear.

The microphone 170C, also called "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can make a sound near the microphone 170C through the human mouth, and input the sound signal into the microphone 170C. The electronic device 100 may be provided with at least one microphone 170C. In other embodiments, the electronic device 100 may be provided with two microphones 170C, which can implement a noise reduction function in addition to collecting sound signals. In other embodiments, the electronic device 100 may further be provided with three, four or more microphones 170C to collect sound signals, reduce noise, identify sound sources, and implement directional recording functions.

The earphone jack 170D is used to connect wired earphones. The earphone interface 170D may be the USB interface 130, or may be a 3.5mm open mobile terminal platform (OMTP) standard interface, a cellular telecommunications industry association of the USA (CTIA) standard interface.

The keys 190 include a power-on key, a volume key, and the like. Keys 190 may be mechanical keys. It can also be a touch key. The electronic device 100 may receive key inputs and generate key signal inputs related to user settings and function control of the electronic device 100 .

Motor 191 can generate vibrating cues. The motor 191 can be used for vibrating alerts for incoming calls, and can also be used for touch vibration feedback. For example, touch operations acting on different applications (such as taking pictures, playing audio, etc.) can correspond to different vibration feedback effects. The motor 191 can also correspond to different vibration feedback effects for touch operations on different areas of the display screen 194 . Different application scenarios (for example: time reminder, receiving information, alarm clock, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also support customization.

The indicator 192 can be an indicator light, which can be used to indicate the charging state, the change of the power, and can also be used to indicate a message, a missed call, a notification, and the like.

The SIM card interface 195 is used to connect a SIM card. The SIM card can be contacted and separated from the electronic device 100 by inserting into the SIM card interface 195 or pulling out from the SIM card interface 195 . The electronic device 100 may support 1 or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 195 can support Nano SIM card, Micro SIM card, SIM card and so on. Multiple cards can be inserted into the same SIM card interface 195 at the same time. The types of the plurality of cards may be the same or different. The SIM card interface 195 can also be compatible with different types of SIM cards. The SIM card interface 195 is also compatible with external memory cards. The electronic device 100 interacts with the network through the SIM card to implement functions such as call and data communication. In some embodiments, the electronic device 100 employs an eSIM, ie: an embedded SIM card. The eSIM card can be embedded in the electronic device 100 and cannot be separated from the electronic device 100 .

An embodiment of the present application provides a non-transitory computer-readable storage medium, where the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions cause the computer to execute the embodiments shown in FIG. 1 to FIG. 4 of the present application Provided methods of operation of electronic equipment.

The above-described non-transitory computer-readable storage media may employ any combination of one or more computer-readable media. The computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or a combination of any of the above. More specific examples (non-exhaustive list) of computer readable storage media include: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read only memory , ROM), erasable programmable read only memory (EPROM) or flash memory, optical fiber, portable compact disk read only memory (CD-ROM), optical storage device, magnetic storage device, or any of the above suitable combination. In this document, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

A computer-readable signal medium may include a propagated data signal in baseband or as part of a carrier wave, with computer-readable program code embodied thereon. Such propagated data signals may take a variety of forms including, but not limited to, electromagnetic signals, optical signals, or any suitable combination of the foregoing. A computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device .

Program code embodied on a computer readable medium may be transmitted using any suitable medium including, but not limited to, wireless, wire, optical fiber cable, radio frequency (RF), etc., or any suitable combination of the foregoing.

Computer program code for performing the operations of the present application may be written in one or more programming languages, including object-oriented programming languages—such as Java, Smalltalk, C++, but also conventional Procedural programming language - such as the "C" language or similar programming language. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer, or entirely on the remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any kind of network, including a local area network (LAN) or wide area network (WAN), or it can be connected to an external computer (e.g. use an internet service provider to connect via the internet).

The foregoing describes specific embodiments of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that in the embodiments and still achieve desirable results. Additionally, the processes depicted in the figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

In the description of this application, references to the terms "one embodiment," "some embodiments," "example," "specific example," or "some examples", etc. refer to the specific features described in connection with the embodiment or example. , structure, material or feature is included in at least one embodiment or example of the present application. In this application, schematic representations of the above terms are not necessarily directed to the same embodiment or example. Furthermore, the particular features, structures, materials or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. Furthermore, those skilled in the art may combine and combine the different embodiments or examples described in this application, as well as the features of the different embodiments or examples, without conflicting each other.

In addition, the terms "first" and "second" are only used for descriptive purposes, and should not be construed as indicating or implying relative importance or implying the number of indicated technical features. Thus, a feature delimited with "first", "second" may expressly or implicitly include at least one of that feature. In the description of the present application, "plurality" means at least two, such as two, three, etc., unless expressly and specifically defined otherwise.

Any process or method description in the flowcharts or otherwise described herein may be understood to represent a module, segment or portion of code comprising one or more executable instructions for implementing custom logical functions or steps of the process , and the scope of the preferred embodiments of the present application includes alternative implementations in which the functions may be performed out of the order shown or discussed, including performing the functions substantially concurrently or in the reverse order depending upon the functions involved, which should It is understood by those skilled in the art to which the embodiments of the present application belong.

Depending on the context, the word "if" as used herein can be interpreted as "at" or "when" or "in response to determining" or "in response to detecting." Similarly, the phrases "if determined" or "if detected (the stated condition or event)" can be interpreted as "when determined" or "in response to determining" or "when detected (the stated condition or event)," depending on the context )" or "in response to detection (a stated condition or event)".

It should be noted that the terminals involved in the embodiments of the present application may include, but are not limited to, a personal computer (personal computer, PC), a personal digital assistant (personal digital assistant, PDA), a wireless handheld device, a tablet computer (tablet computer), Mobile phones, MP3 players, MP4 players, etc.

In the several embodiments provided in this application, it should be understood that the disclosed system, apparatus and method may be implemented in other manners. For example, the apparatus embodiments described above are only illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components may be combined. Either it can be integrated into another system, or some features can be omitted, or not implemented. On the other hand, the shown or discussed mutual coupling or direct coupling or communication connection may be through some interfaces, indirect coupling or communication connection of devices or units, and may be in electrical, mechanical or other forms.

In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically alone, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware, or may be implemented in the form of hardware plus software functional units.

The above-mentioned integrated units implemented in the form of software functional units can be stored in a computer-readable storage medium. The above-mentioned software functional unit is stored in a storage medium, and includes several instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) or a processor (processor) to execute the methods described in the various embodiments of the present application. some steps. The aforementioned storage medium includes: U disk, removable hard disk, read only memory (ROM), random access memory (RAM), magnetic disk or optical disk and other media that can store program codes.

The above descriptions are only preferred embodiments of the present application, and are not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application shall be included in the present application. within the scope of protection.

Claims (14)

1. A method of operating an electronic device, comprising:

   Obtain pupil area information of users using electronic devices;

   determining the display position of the target button icon in the current display interface of the electronic device according to the pupil area information of the user;

   generating the target button icon, and displaying the target button icon at the display position;

   detecting the gaze duration of the user on the target button icon;

   According to the gaze duration, determine the operation weight for the target button icon;

   An operation instruction corresponding to the operation weight is determined, and the operation instruction is executed.
2. The method according to claim 1, wherein the determining the display position of the target button icon in the current display interface of the electronic device according to the pupil area information of the user comprises:

   According to the pupil area information of the user, a pre-trained machine learning model is used to obtain the display position of the target button icon in the display interface.
3. The method according to claim 2, wherein the acquiring the pupil area information of the user using the electronic device comprises:

   Capture the area where the pupil of the user is located through a sensor in the electronic device;

   Through the collector in the electronic device, the pupil is marked in the area where the pupil is located, and the pupil area information of the user is acquired.
4. The method according to claim 2, wherein the obtaining the display position of the target button icon in the display interface by using a pre-trained machine learning model according to the pupil area information of the user comprises:

   According to the pupil area information of the user, calculate the distance of the center point of the pupil of the user relative to the horizontal direction of the sensor, and the angle of the center point of the pupil of the user relative to the vertical direction of the sensor;

   According to the distance and the angle, a pre-trained machine learning model is used to obtain the display position of the target button icon in the display interface.
5. The method according to any one of claims 1-4, wherein the determining the operation instruction corresponding to the operation weight comprises:

   When the operation weight is the first level, determine that the operation instruction corresponding to the operation weight is the operation instruction corresponding to the click operation;

   When the operation weight is the second level, determine that the operation instruction corresponding to the operation weight is the operation instruction corresponding to the long-press operation;

   When the operation weight is the first level combined with the second level, it is determined that the operation instruction corresponding to the operation weight is the operation instruction corresponding to the sliding operation.
6. The method according to any one of claims 1-4, wherein after determining the operation weight for the target button icon according to the gaze duration, the method further comprises:

   According to the operation weight, the display effect of the target button icon is adjusted; wherein, the display effect includes color, size and/or brightness.
7. An operating device for electronic equipment, comprising:

   an acquisition module, used to acquire the pupil area information of the user who uses the electronic device;

   a determination module, configured to determine the display position of the target button icon in the current display interface of the electronic device according to the pupil area information of the user;

   a generating module for generating the target button icon;

   a display module, configured to display the target button icon generated by the generation module at the display position determined by the determination module;

   a detection module, configured to detect the gaze duration of the user on the target button icon;

   The determining module is further configured to determine an operation weight for the target button icon according to the gaze duration; and determine an operation instruction corresponding to the operation weight;

   The execution module is used for executing the operation instruction.
8. The device of claim 7, wherein:

   The determining module is specifically configured to obtain the display position of the target button icon in the display interface by using a pre-trained machine learning model according to the pupil area information of the user.
9. The device of claim 8, wherein:

   The acquisition module is specifically configured to capture the area where the pupil of the user is located through a sensor in the electronic device; and through the collector in the electronic device, mark the pupil in the area where the pupil is located and acquire the pupil of the user Regional information.
10. The device according to claim 8, wherein the determining module comprises:

    a calculation submodule, configured to calculate the distance of the user's pupil center point relative to the horizontal direction of the sensor and the angle of the user's pupil center point relative to the vertical direction of the sensor according to the pupil area information of the user;

    The position determination submodule is configured to obtain the display position of the target button icon in the display interface by using a pre-trained machine learning model according to the distance and the angle.
11. The device according to any one of claims 7-10, wherein the determining module comprises:

    an instruction determination submodule, configured to determine that the operation instruction corresponding to the operation weight is an operation instruction corresponding to the click operation when the operation weight is the first level; when the operation weight is the second level, determine that the operation instruction corresponding to the operation weight is the operation instruction corresponding to the click operation The operation instruction corresponding to the operation weight is the operation instruction corresponding to the long-press operation; when the operation weight is the first level combined with the second level, the operation instruction corresponding to the operation weight is determined to be the operation instruction corresponding to the sliding operation.
12. The device according to any one of claims 7-10, further comprising:

    an adjustment module, configured to adjust the display effect of the target button icon according to the operation weight after the determination module determines the operation weight for the target button icon; wherein the display effect includes color, size and/or or brightness.
13. An electronic device comprising:

    at least one processor; sensors and collectors communicatively coupled to the processor; and

    at least one memory communicatively coupled to the processor, wherein:

    The memory stores program instructions executable by the processor, and the processor invokes the program instructions to be able to perform the method as claimed in any one of claims 1 to 6 .
14. A non-transitory computer-readable storage medium storing computer instructions that cause the computer to perform the method of any one of claims 1 to 6.

Images