WO2022001564A1

WO2022001564A1 - Operation set obtaining and executing methods and apparatuses, storage medium, and terminal device

Info

Publication number: WO2022001564A1
Application number: PCT/CN2021/097922
Authority: WO
Inventors: 高宏华
Original assignee: 中兴通讯股份有限公司
Priority date: 2020-06-30
Filing date: 2021-06-02
Publication date: 2022-01-06
Also published as: CN113946257A

Abstract

The present invention provides operation set obtaining and executing methods and apparatuses, a storage medium, and a terminal device. The obtaining method comprises: receiving one or more operations for the terminal device, and obtaining operation information of each operation in the one or more operations, wherein the operation information comprises: order identification information for identifying an operation order of an operation in the one or more operations, and operation description data of the operation; and generating an operation set according to the operation information, wherein the operation set comprises: the operation information of the one or more operations. According to the present invention, recording of a series of operations of a user is achieved, so that the terminal device can automatically execute a series of operations according to the recorded operation set in the case of a user trigger or satisfying execution conditions, and the problem of how to simplify user operations on the terminal device is solved.

Description

Method and apparatus for obtaining and executing operation set, storage medium and terminal device

technical field

The present disclosure relates to the field of communications, and in particular, to a method and apparatus for acquiring and executing an operation set, a storage medium, and a terminal device.

Background technique

With the improvement of the functions of terminal devices (eg, mobile phones, tablet computers, notebook computers, personal computers (Personal Computer, PC for short), etc.), the use and operations thereof are becoming more and more complicated.

To give a simple example, when the user needs to use the shared bicycle business, the user needs to perform a series of operations continuously, including: running the shared bicycle software or running the multi-service software including the shared bicycle function and clicking to enter the bicycle business, open the data flow, open the Locate, turn on bluetooth, and then click Scan bicycles to let the phone enter the state of scanning bicycles. Such operations are too cumbersome for users, and too complicated for groups such as the elderly.

With the emergence of more and more intelligent services and the increasingly diverse and complex needs of users, in order to meet user needs, the user operations that need to be performed on terminal devices are becoming more and more complex. User operation is an urgent problem to be solved at present.

SUMMARY OF THE INVENTION

Embodiments of the present disclosure provide a method and apparatus for acquiring and executing an operation set, a storage medium, and a terminal device, so as to at least solve the problem of how to simplify user operations on the terminal device.

According to some embodiments of the present disclosure, a method for obtaining an operation set is provided, including: receiving one or more operations on a terminal device, and obtaining operation information of each of the one or more operations, wherein: The operation information includes: sequence identification information used to identify the operation sequence of the operation in the one or more operations, and operation description data of the operation; an operation set is generated according to the operation information, wherein the operation set Including: the operation information of the one or more operations.

According to some embodiments of the present disclosure, there is provided a method for executing an operation set, comprising: obtaining the operation set in the case of receiving an operation set execution request corresponding to the operation set or judging that the execution condition corresponding to the operation set is satisfied, Wherein, the operation set includes: operation information of one or more operations, the operation information includes: sequence identification information used to identify the operation sequence of the operation in the one or more operations, an operation description of the operation data; and according to the operation sequence identified by the sequence identification information, perform the one or more operations according to the operation description data.

According to some embodiments of the present disclosure, an apparatus for obtaining an operation set is provided, including: a first obtaining module configured to receive one or more operations on a terminal device, and obtain each of the one or more operations Operation information of the operation, wherein the operation information includes: sequence identification information for identifying the operation sequence of the operation in the one or more operations, and operation description data of the operation; The operation information generates an operation set, wherein the operation set includes: the operation information of the one or more operations.

According to some embodiments of the present disclosure, an apparatus for executing an operation set is provided, including: a second obtaining module, configured to: when an operation set execution request corresponding to the operation set is received or an execution condition corresponding to the operation set is judged to be satisfied , obtain the operation set, wherein the operation set includes: operation information of one or more operations, and the operation information includes: sequence identification information used to identify the operation sequence of the operation in the one or more operations , the operation description data of the operation; the execution module is configured to execute the one or more operations according to the operation description data according to the operation sequence identified by the sequence identification information.

According to some embodiments of the present disclosure, a computer-readable storage medium is also provided, and a computer program is stored in the computer-readable storage medium, wherein the computer program is configured to execute any one of the above method implementations when running steps in the example.

According to some embodiments of the present disclosure, there is also provided a terminal device including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute any one of the above methods steps in the examples.

Through the embodiments of the present disclosure, since one or more operations on the terminal device can be automatically received, the operation information of each operation in the one or more operations can be acquired, and an operation set can be generated according to the operation information, so that the automatic recording can be performed. A series of operations performed by the user, so that the terminal device can automatically perform a series of operations according to the recorded operation set when triggered by the user or satisfying the execution conditions. Therefore, the problem of how to simplify the user operations on the terminal device can be solved, and the Custom shortcuts are recorded and executed.

Description of drawings

Fig. 1 is a hardware structure block diagram of a mobile terminal of a method for acquiring and executing an operation set;

Fig. 2 is the flow chart of the acquisition method of operation set;

Fig. 3 is the flow chart of the execution method of operation set;

Fig. 4 is the structural block diagram of the acquisition device of operation set;

Fig. 5 is the structural block diagram of the execution apparatus of the operation set;

Fig. 6 is the schematic diagram that application coordinate moves;

FIG. 7 is a schematic diagram of different switch states.

detailed description

Faced with the problem of how to simplify user operations on the terminal device, it may be considered to simplify user operations to a certain extent by setting shortcuts. Currently on a smartphone, if a user wants to set a shortcut operation, they can long press on the desktop and select "Add Widget" in the pop-up menu, find "Set Shortcut", and find the corresponding setting item in it to create a related Shortcut, this function is convenient for users to quickly call a setting item for setting operation. The more common way to set a shortcut on PC is to right-click and select "Send to Desktop Shortcut", and then a shortcut can be generated on the desktop, which is convenient for users to quickly reach a certain location or run a certain software.

These shortcuts have the following disadvantages:

(1) The functions of these shortcuts are too simple. For example, the shortcuts on the PC are either to a certain folder location or to start an application; while the shortcuts on the smartphone, only a fixed number of setting items can allow Users set shortcuts, which are rarely used due to the poor practicality of this function.

(2) These shortcuts are too solid. Whether it is a PC or a smartphone, the shortcuts are preset after leaving the factory. The user can only choose to use or not to use, the function is fixed, and the user cannot customize the function of the shortcut at all.

(3) These shortcuts cannot be started regularly, nor can they be sent from user A to user B.

However, when a user uses a terminal device, more complex operations and continuous operations are often generated according to personal usage habits. For example, after going to work, the user may open the mailbox, notepad, various work-related tool software, etc. after turning on the computer, so that the computer can enter the working state; for another example, when the user wants to use the shared bicycle, he often needs to run the shared bicycle software or run The multi-service software including the shared bicycle function and click to enter the bicycle business, turn on the data flow, turn on the positioning, turn on the Bluetooth, and then click on the scan bicycle, so that the mobile phone can enter the state of scanning the bicycle. Obviously, the shortcut technology mentioned above cannot meet the needs of different users.

In order to solve the above problems, the embodiments of the present disclosure provide a solution that allows users to customize shortcuts according to their own usage habits and preferences, the solution can save the user's operation collection, and generate shortcuts for users to use in subsequent use. Can operate quickly. In addition, the solution also supports sending the shortcut to other users for use, or adding a timer for regular execution to derive more powerful functions. This solution can be used in a wide range of scenarios, such as: mobile phone one-click navigation, one-click scanning of bicycles; after the computer is turned on, let the computer automatically open various software to be opened, and enter the working mode; for remote assistance (such as remote setting an alarm clock), backup Waiting for a series of operations; timing check-in and so on.

Embodiments of the present disclosure are hereinafter described in detail with reference to the accompanying drawings and in conjunction with the description of some examples.

It should be noted that the terms "first", "second" and the like in the description and claims of the present disclosure and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific sequence or sequence.

The method embodiments provided in the embodiments of the present disclosure may be executed in a mobile terminal, a computer terminal, or a similar terminal device. Taking running on a mobile terminal as an example, FIG. 1 is a block diagram of the hardware structure of a mobile terminal with a method for acquiring and executing an operation set. As shown in FIG. 1 , the mobile terminal may include one or more (only one is shown in FIG. 1 ) processor 102 (the processor 102 may include, but is not limited to, a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data, wherein the above-mentioned mobile terminal may also include a transmission device 106 and an input and output device 108 for communication functions. Those of ordinary skill in the art can understand that the structure shown in FIG. 1 is only a schematic diagram, which does not limit the structure of the above-mentioned mobile terminal. For example, the mobile terminal may also include more or fewer components than those shown in FIG. 1 , or have a different configuration than that shown in FIG. 1 .

The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as computer programs corresponding to the acquisition and execution methods of the operation sets in the embodiments of the present disclosure. The processor 102 runs the computer programs stored in the memory 104 by running the computer programs , so as to perform various functional applications and data processing, that is, to implement the above method. Memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory located remotely from the processor 102, and these remote memories may be connected to the mobile terminal through a network. Examples of such networks include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

Transmission means 106 are used to receive or transmit data via a network. The specific example of the above-mentioned network may include a wireless network provided by a communication provider of the mobile terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, NIC for short), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 may be a radio frequency (Radio Frequency, RF for short) module, which is used to communicate with the Internet in a wireless manner.

Some embodiments of the present disclosure provide a method for obtaining an operation set running on the above-mentioned terminal device. Fig. 2 is a flowchart of the method for obtaining an operation set. As shown in Fig. 2 , the flow process includes the following steps:

Step S202: Receive one or more operations on the terminal device, and acquire operation information of each operation in the one or more operations, wherein the operation information includes: an operation information used to identify that the operation is performed in the one or more operations. Sequence identification information of the operation sequence in the operation, operation description data of the operation;

Step S204, generating an operation set according to the operation information, wherein the operation set includes: the operation information of the one or more operations.

Through the above steps, since one or more operations on the terminal device can be automatically received, the operation information of each operation in the one or more operations can be acquired, and an operation set can be generated according to the operation information, so that the user's operations can be automatically recorded. A series of operations, so that the terminal device can automatically execute a series of operations according to the recorded operation set when the user triggers or meets the execution conditions. Therefore, it can solve the problem of how to simplify the user operation on the terminal device and realize the customization record of shortcuts.

Wherein, the execution subject of the above steps may be a terminal device or the like, but is not limited thereto.

In at least one exemplary embodiment, the operation information may further include: a relevant frame image corresponding to the operation, wherein the relevant frame image includes: a valid frame image before execution, a valid frame image during execution, and a valid frame image after execution frame image. The pre-execution valid frame image may be a screen image at a first predetermined time (eg, 30ms) before the operation is performed, and the post-execution valid frame image may be a screen image at a second predetermined time (eg, 80ms) after the operation is performed. For some transient operations (for example, clicking on a webpage, entering a function page in an APP, etc.), the effective frame image after the execution of the previous operation may be the same as the effective frame image before the execution of the next operation; Operations that last for a certain period of time (for example, making a call), the effective frame image after the execution of the previous operation (the call page image after the call is made) and the effective frame image before the execution of the next operation (the page image after the call is hung up). ) may be different.

In some exemplary embodiments, the execution-time valid frame image can assist the acquisition of the operation description data. In the case that the complete operation description data cannot be obtained through the system's direct reading operation, the execution-time valid frame image can be obtained by The image performs image recognition to obtain the operation description data of the operation.

In at least one exemplary embodiment, step S202 may include the following operations:

Receive an operation collection collection request;

In response to the received operation set collection request, the terminal device obtains the relevant frame image corresponding to each operation in the one or more operations through a screen recording function or a screen capture function, and collects the one or more operations The sequence identification information and the operation description data of each operation in the . until a collection end indication is received. In some exemplary embodiments, in response to the received operation set collection request, the screen of the terminal device may be controlled to display an initial page, and each of the one or more operations may be acquired through a screen recording function or a screen capture function. The corresponding relevant frame images are operated, and the sequence identification information and the operation description data of each operation in the one or more operations are collected until a collection end indication is received.

Through the above solution, the user can collect a request by operating a collection (for example, by clicking a control for recording a shortcut on the operation interface to issue the request), and initiate the recording process of the shortcut, and the terminal device can obtain it by recording a screen or taking a screenshot. To the relevant frame image corresponding to the operation, the sequence identifier of each operation in the one or more operations can be obtained through direct reading by the system, or direct reading by the system in combination with the image recognition of the valid frame image at the time of execution. information and data describing the operation.

In at least one exemplary embodiment, the sequence identification information for identifying the operation sequence of the operation in the one or more operations may include at least one of the following: the operation time of the operation, the operation time of the operation in the one or more operations. or the sequence number of the operation in multiple operations.

In at least one exemplary embodiment, the operation description data of the operation may include at least one of the following: an operation category, a coordinate parameter, a duration parameter, key identification information, identification information of a sensor that collects biometrics, Collection parameters, the operation object corresponding to the operation, the description information of the execution page, and the description information of the result page. The operation category may include, but is not limited to, at least one of the following: clicking on the screen, sliding the screen, pressing a button, and collecting biological features.

With different operation categories, the collected operation description data may also be different, and the content of the specific operation description data can be set according to actual needs. For example, when the operation category includes the click screen, the operation The description data may include at least one of the following: the coordinates of the click screen, the duration, the operation object corresponding to the operation, the description information of the execution page, and the description information of the result page; when the operation category includes the sliding screen, all The operation description data may include at least one of the following: the starting coordinates of the sliding screen, the ending coordinates of the sliding screen, the duration, the operation object corresponding to the operation, the description information of the execution page and the description information of the result page; in the operation category In the case where the pressed key is included, the operation description data may include at least one of the following: key identification information for identifying the pressed key, duration, operation object corresponding to the operation, execution page description information, and Result page description information; in the case where the operation category includes the collection of biometric features, the operation description data may include at least one of the following: sensor identification information and collection parameters of the sensor used to collect the biometric feature, the operation Corresponding operation object, execution page description information, and result page description information.

In at least one exemplary embodiment, the operation category may be acquired based on a screen touch signal, a key touch signal, or a system sensor call signal; and/or,

The coordinate parameters may be obtained based on a screen touch signal; and/or,

The duration parameter may be obtained based on a screen touch signal; and/or,

The key identification information may be obtained based on a key touch signal; and/or,

The identification information and the acquisition parameters of the sensors that collect biological features may be obtained based on the system sensor call signal; and/or,

The operation object and execution page description information corresponding to the operation can be obtained based on the image recognition technology according to the valid frame image corresponding to the execution time of the operation, or based on the image recognition technology based on the valid frame image before execution corresponding to the operation combined with the coordinates Parameter acquisition, for example, since the effective frame image often displays the area currently being operated with visual effects visible to the naked eye during execution, such as which function button to click, which slider to slide, etc. The valid frame image identifies the operation object corresponding to the current operation, and further identifies the execution page description information corresponding to the current operation, such as the text on the function button, the text description around the slider, etc.; It can be implemented based on the valid frame image before execution. The only difference is that the valid frame image before execution needs to be combined with the coordinate parameters of the operation to get which operation object the user is currently operating, and further identify the operation object or the operation object. description information surrounding the execution page; and/or,

The description information of the result page corresponding to the operation can be obtained from the valid frame image after execution corresponding to the operation based on image recognition technology. For example, the page description information can be identified in the valid frame image after execution corresponding to the operation based on the image recognition technology. As the result page description information, preferably, a result description keyword may be further identified in the identified page description information according to an algorithm obtained by machine learning as the result page description information.

In at least one exemplary embodiment, after step S204, the method may further include at least one of the following:

A. Save the set of operations;

B. After setting the execution condition corresponding to the operation set, save the operation set and the execution condition corresponding to the operation set;

C. Send the set of operations;

D. After setting the execution condition corresponding to the operation set, send the operation set and the execution condition corresponding to the operation set.

In the above manner, the execution condition corresponding to the operation set can be set, and the execution condition may include execution time, pre-event or remote trigger, etc., thereby realizing more flexible shortcut triggering. In addition, the operation set can also be sent to other terminal devices, so as to realize remote control guidance to other terminal devices.

Some embodiments of the present disclosure provide a method for executing an operation set running on the above-mentioned terminal device. FIG. 3 is a flowchart of the method for executing an operation set. As shown in FIG. 3 , the process includes the following steps:

Step S302, in the case of receiving the operation set execution request corresponding to the operation set or judging that the execution condition corresponding to the operation set is satisfied, obtain the operation set, wherein the operation set includes: operation information of one or more operations, The operation information includes: sequence identification information for identifying an operation sequence of the operation in the one or more operations, and operation description data of the operation;

Step S304, according to the operation sequence identified by the sequence identification information, perform the one or more operations according to the operation description data.

Through the above steps, because the operation set can be automatically responded to the execution request of the operation set or when it is judged that the execution condition is satisfied, the operation set can be obtained and the operation sequence identified by the sequence identification information can be identified, and the one or more operations can be executed according to the operation description data. Therefore, the terminal device can automatically perform a series of operations according to the shortcut, which solves the problem of how to simplify user operations on the terminal device and realizes the execution of the customized shortcut.

In some exemplary embodiments, the execution-time valid frame image may assist the acquisition of the operation description data. In the case that the content of the operation description data is incomplete and the operation cannot be performed accurately, the execution-time valid frame image may be processed by Image recognition to obtain the complete operation description data of the operation.

In at least one exemplary embodiment, step S304 may include:

(1) Determine the current operation to be performed according to the sequence identification information;

(2) according to the current screen image before the execution and the corresponding effective frame image before the execution of the current operation, determine whether the preconditions for executing the current operation are satisfied, and if the current operation is satisfied, the current operation is performed;

(3) Determine whether the current operation is successfully executed, and if the execution is successful, continue to determine and execute the next current operation to be executed until the one or more operations are executed.

In at least one exemplary embodiment, step (2) may include the following processing:

determining whether the current screen image before execution includes the operation object corresponding to the current operation;

If the current pre-execution screen image includes an operation object corresponding to the current operation, in the case where the position of the operation object corresponding to the current operation is the same in the current pre-execution screen image and the pre-execution effective frame image In the next step, the current operation is performed according to the operation description data of the current operation; and/or, in the screen image before the current execution and the effective frame image before the execution of the operation object corresponding to the current operation When the position changes, adjust the operation description data of the current operation according to the position of the operation object corresponding to the current operation in the screen image before the current operation, and adjust the operation description data of the current operation according to the adjusted value of the current operation. The operation description data performs the current operation.

In at least one exemplary embodiment, the operation object corresponding to the current operation based on the above process may be included in the operation description data of the current operation, or corresponding to the current operation based on an image recognition technology The valid frame image at the time of execution is obtained, or based on the image recognition technology, the valid frame image before execution corresponding to the current operation is obtained in combination with the coordinate parameters of the current operation.

Through the above process, it is possible to implement a pre-check before each operation is performed, so as to decide whether to perform the operation directly or to perform the operation after adjustment and error correction.

In at least one exemplary embodiment, step (1) may include at least one of the following:

Identify, based on image recognition technology, whether the current screen image before execution includes an icon of an operation object corresponding to the current operation (which may include application icons, in-app control icons, and other icons on which users can perform operations, including but not (limited to icons that can be clicked, swiped or performed other on-screen operations), according to the recognition result, determine whether the current pre-execution screen image includes the operation object corresponding to the current operation. The technology identifies whether the application icon of the bicycle APP is included in the screen image before the current execution;

Identify the page description information included in the current pre-execution screen image based on image recognition technology, match the identified page description information with the execution page description information corresponding to the current operation, and determine the current pre-execution page description information according to the matching result. Whether the screen image includes the operation object corresponding to the current operation. For example, for an operation to start a bicycle APP, the description information of the execution page corresponding to the current operation can be "bicycle", and the screen image before the current execution can be identified based on image recognition technology. Whether there is the word "Bike" in the screen, if so, it is judged to be a match. In addition, if there is a word "Bike", "Bicycle" or "Danche" in the screen image before the current execution based on image recognition technology, it is also considered to be a match. The same APP may have different language versions, and there is a corresponding relationship between the languages. Based on this correspondence, the matching of multiple language versions of APPs can be realized, and there will be no operation on the Chinese APP when entering, and the execution terminal is the English version. The system causes the situation that the name of the APP cannot be recognized by an English name, which is more intelligent.

In at least one exemplary embodiment, the execution page description information corresponding to the current operation based on the above process may be included in the operation description data of the current operation, or based on image recognition technology according to the The valid frame image at the time of execution corresponding to the current operation is obtained, or based on the image recognition technology, the valid frame image before execution corresponding to the current operation is obtained in combination with the coordinate parameters of the current operation.

In at least one exemplary embodiment, adjusting the operation description data of the current operation according to the position of the operation object corresponding to the current operation in the screen image before the current execution may include one of the following:

In the case that the coordinate parameters of the operation object in the screen image before the current execution are consistent with the coordinate parameters of the click on the operation object on the screen, the coordinate parameters in the operation description data of the current operation are changed to Replaced with the coordinate parameters of the operation object corresponding to the current operation in the screen image before the current execution;

In the case that the coordinate parameters of the operation object in the screen image before the current execution are inconsistent with the coordinate parameters of clicking the operation object on the screen, the coordinates of the operation object after the click changes position are determined according to the following formula Parameter (xd2, yd2): (x1, y1)/(xd1, yd1)=(x2, y2)/(xd2, yd2), where (x1, y1) is the valid frame of the operation object before the execution The coordinate parameters in the image, (x2, y2) are the coordinate parameters of the operation object in the screen image before the current execution, and (xd1, yd1) are the coordinate parameters in the operation description data before adjustment.

In at least one exemplary embodiment, in the case that the current pre-execution screen image does not include an operation object corresponding to the current operation, the method further includes one of the following:

A. Confirm the failure to perform the one or more operations;

B. Return to repeat the previous operation of the current operation;

C. Prompt the user to continue to perform the unfinished operation in the one or more operations;

D. Prompt the user to perform the current operation, and after the current operation is completed, continue to follow the sequence of operations identified by the sequence identification information, and perform the one or more operations not completed according to the operation description data operation.

Through this solution, in the pre-check before each operation is performed, if it is found that the icon does not exist and the operation cannot be automatically realized, methods such as retry, transfer to the user to help perform the current operation or all subsequent operations, or failure to be declared can be adopted.

In at least one exemplary embodiment, determining whether the current operation is successfully performed in step (3) may include at least one of the following:

Identify whether the screen image after the current execution is consistent with the effective frame image after execution corresponding to the current operation based on image recognition technology, and determine that the current operation is successfully executed if they are consistent;

Identify, based on image recognition technology, whether the current screen image includes an icon of an operation object corresponding to the next operation of the current operation, and if it is included, determine that the current operation is successfully executed;

Identify the page description information included in the currently executed screen image based on the image recognition technology, determine whether the identified page description information matches the result page description information corresponding to the current operation, and determine the current The operation performed successfully.

Through this method, the inspection after the execution of each operation step can be carried out, thereby realizing more effective and accurate operation flow control.

In at least one exemplary embodiment, the result page description information corresponding to the current operation used in the above process may be included in the operation description data of the current operation, or based on image recognition technology according to the current operation The post-execution valid frame image corresponding to the operation is acquired.

In at least one exemplary embodiment, before step S302, the method may further include one of the following:

A. Receive the one or more operations on the terminal device, obtain operation information of each operation in the one or more operations, generate the operation set according to the operation information, and save the operation set;

B. Receive the operation set sent by other terminal equipment;

C. Receive the one or more operations on the terminal device, and obtain the operation information of each operation in the one or more operations, generate the operation set according to the operation information, and set the corresponding operation set of the operation set. the execution condition, and save the operation set and the execution condition corresponding to the operation set;

D. Receive the operation set sent by other terminal equipment and the execution condition corresponding to the operation set.

From the description of the above embodiments, those skilled in the art can clearly understand that the method according to the embodiment of the present disclosure can be implemented by means of software plus a necessary general hardware platform, and certainly can also be implemented by hardware, but in many cases the former is a better implementation. Based on this understanding, the technical solutions of the present disclosure essentially or the parts that contribute to the prior art can be embodied in the form of software products, and the computer software products are stored in a storage medium (such as ROM/RAM, magnetic disk, CD-ROM), including several instructions to make a terminal device (which may be a mobile phone, a computer, a server, or a network device, etc.) to execute the method described in the embodiments of the present disclosure.

Some embodiments of the present disclosure provide an apparatus for acquiring an operation set, the apparatus is used to implement the embodiments and preferred implementations of the above-mentioned method for acquiring an operation set, which have been described and will not be repeated. As used below, the term "module" may be a combination of software and/or hardware that implements a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, implementations in hardware, or a combination of software and hardware, are also possible and contemplated.

Fig. 4 is a structural block diagram of an apparatus for obtaining an operation set. As shown in Fig. 4, the apparatus includes:

The first obtaining module 42 is configured to receive one or more operations on the terminal device, and obtain operation information of each operation in the one or more operations, wherein the operation information includes: an operation information used to identify where the operation is performed. Sequence identification information of the sequence of operations in the one or more operations, and operation description data of the operations;

The generating module 44 is configured to generate an operation set according to the operation information, wherein the operation set includes: the operation information of the one or more operations.

With the above device, one or more operations on the terminal device can be automatically received, operation information of each operation in the one or more operations can be acquired, and an operation set can be generated according to the operation information, so that the user's operations can be automatically recorded. A series of operations, so that the terminal device can automatically execute a series of operations according to the recorded operation set when the user triggers or meets the execution conditions. Therefore, it can solve the problem of how to simplify the user operation on the terminal device and realize the customization record of shortcuts.

Wherein, the above-mentioned apparatus may be provided in a terminal device, but is not limited to this.

It should be noted that the above modules can be implemented by software or hardware, and the latter can be implemented in the following ways, but not limited to this: the above modules are all located in the same processor; or, the above modules can be combined in any combination The forms are located in different processors.

Some embodiments of the present disclosure provide an apparatus for executing an operation set, and the apparatus is used to implement the embodiments and preferred implementations of the above-mentioned method for executing an operation set, which have been described and will not be repeated. As used below, the term "module" may be a combination of software and/or hardware that implements a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, implementations in hardware, or a combination of software and hardware, are also possible and contemplated.

Fig. 5 is the structural block diagram of the execution apparatus of the operation set, as shown in Fig. 5, this apparatus includes:

The second obtaining module 52 is configured to obtain the operation set when receiving the operation set execution request corresponding to the operation set or judging that the execution condition corresponding to the operation set is satisfied, wherein the operation set includes: one or more Operation information of the operation, the operation information includes: sequence identification information for identifying the operation sequence of the operation in the one or more operations, and operation description data of the operation;

The execution module 54 is configured to execute the one or more operations according to the operation description data according to the operation sequence identified by the sequence identification information.

Through the above device, because the operation set can be automatically responded to the execution request of the operation set or when it is judged that the execution condition is satisfied, the operation set can be obtained and the operation sequence identified by the sequence identification information can be obtained, and the one or more operations can be executed according to the operation description data. Therefore, the terminal device can automatically perform a series of operations according to the shortcut, which solves the problem of how to simplify user operations on the terminal device and realizes the execution of the customized shortcut.

Embodiments of the present disclosure also provide a computer-readable storage medium, where a computer program is stored in the computer-readable storage medium, wherein the computer program is configured to execute the steps in the method embodiments when running.

In some exemplary embodiments, the above-mentioned computer-readable storage medium may include, but is not limited to, a USB flash drive, a read-only memory (Read-Only Memory, referred to as ROM for short), and a random access memory (Random Access Memory, referred to as RAM for short) ), mobile hard disks, magnetic disks or optical discs and other media that can store computer programs.

Embodiments of the present disclosure also provide a terminal device, including a memory and a processor, where a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in the above method embodiments.

In some exemplary embodiments, the above-mentioned terminal device may further include a transmission device and an input/output device, wherein the transmission device is connected to the above-mentioned processor, and the above-mentioned input/output device is connected to the above-mentioned processor.

For specific examples of the computer-readable storage medium and the terminal device, reference may be made to the examples described in the foregoing embodiments and exemplary implementation manners, and details are not repeated here.

The following takes an example of an implementation of acquiring a user's operation set by recording a screen to generate a shortcut to describe in detail the technical solutions of the method for acquiring and executing the operation set in the embodiment of the present disclosure. It should be noted that the following exemplary embodiments are exemplary implementations of the solutions described in the foregoing embodiments, which may be understood as further explanations for the solutions of the foregoing embodiments, but do not constitute any limitation to the foregoing embodiments. Improperly limited.

Image recognition technology refers to the use of computers to process, analyze and understand images to identify various patterns of targets and objects. It is a practical application of deep learning algorithms. The traditional image recognition process is divided into four steps: image acquisition→image preprocessing→feature extraction→image recognition. Through image recognition technology, the computer can identify the content in the photo. The present exemplary embodiment obtains a screenshot of the user's operation process by means of a screen recording technology, and obtains relevant data of the user's operation by means of an image recognition technology.

In this exemplary embodiment, through the screen recording technology, the user's operation steps are recorded (the user operation description data such as click event set, operation interval, etc. are saved during recording, and the user's operation steps before, during, and after the operation are saved. keyframe images) and store them to generate a sequence of operations in chronological order, which in turn generate corresponding shortcuts.

When the shortcut is executed, the system will execute the pre-stored set of events in chronological order, just like video playback. In particular, when the shortcut performs each step of the user operation, combined with the image recognition technology, the system will compare the image before the operation, the image when the operation is performed, and the image after the operation is performed with the pre-stored images respectively. Yes, specifically, the system will identify the image content before and after the operation through image recognition technology (for example, here is a switch switch, what is the word on the switch; there is an icon, which application is the icon), and the pre-stored image content The key frame images are compared, and according to the result of the image content comparison, it is judged whether the preconditions of the current operation execution are satisfied, whether the current operation is abnormal, and whether the result after execution is normal. If there is a difference between the images before the operation is performed, the preconditions of the operation are not satisfied, the system can perform appropriate error correction according to the image recognition technology; Judgment of the result, if it is judged that the result of the execution of the operation is a failure, the system can perform exception processing.

Further, this shortcut can be sent to other users for use, or a timer can be added for regular execution to derive more powerful functions. This exemplary embodiment can be applied to a variety of application scenarios, for example, one-key navigation on mobile phones, one-key scanning of bicycles; after the computer is turned on, let the computer automatically open various software to be opened, and enter the working mode; remote assistance (such as remote setting Alarm clock), backup a series of operations; timed punching and so on.

This exemplary embodiment mainly involves four modules, which will be described in detail below.

(1) Information collection module: It is mainly responsible for recording the screen, saving the key frame images before, during and after the user's operation, and collecting the data of each operation of the user. For example, the time and category of each operation (such as clicking the screen, volume keys), if the screen is clicked, the coordinates will also be recorded, whether it is a long press or a short press, and so on.

(2) Storage module: used to store the data collected by the information collection module, and its data structure can be a list, similar to that shown in Table 1 below. Here, each operation is numbered in chronological order, with the operation performed first being 0001, followed by 0002, and so on.

Table 1: Data stored by the storage module

(3) Image recognition module: According to the parameters of the operation, combined with the key frame image saved during the operation, the object of the user's operation is recognized. Before each operation, compare the current image with the pre-operation key frame stored in advance, and determine whether there is a difference between the two frame images. For example: whether the operation object is still there; whether the position of the operation object is the same; if there is a difference, notify the control module to correct the error. After each operation, compare the current image with the pre-stored key frames after the operation, and judge whether the current operation is successful according to whether there is a difference between the two frames of images; if the operation fails, notify the control module to handle exceptions.

(4) Control module: When the user records the screen, the control system enters the initial screen recording state, and is responsible for starting the information collection module to collect information, and then stores the collected information in the storage module. After executing the shortcut, execute the pre-stored operation steps in chronological order and relevant parameters, and start the image recognition module to compare the current image data and the pre-stored image data in real time, and judge whether to perform error correction or re-run according to the recognition result. try, handle exceptions, abort the operation, or prompt the user to choose the next operation.

The following describes the implementation process of the embodiment of the present disclosure by taking a scenario in which a user scans a shared bicycle by using the "Bicycle" APP. The process includes an information collection process, a shortcut execution process, an error correction process, and a running result judgment process. Be explained. It should be noted that the following exemplary embodiments are specific implementations of the solutions described in the foregoing embodiments and exemplary embodiments in specific scenarios, which may be understood as solutions to the foregoing embodiments and exemplary embodiments. further explanation, but does not constitute an improper limitation to the foregoing embodiments and exemplary embodiments.

1. The information collection process includes the following steps:

(1) An entry for users to record shortcuts will be added on the interface, and users can start the custom shortcut function through this entry;

(2) The system switches to the initial state of screen recording, generally returning to the initial desktop of the mobile phone;

(3) The information collection module saves a frame of the current screenshot image and waits for user input;

(4) The user runs the "Bicycle" APP and clicks on the screen;

(5) At the same time as the click operation, the information acquisition module saves a frame of the image during the operation, and stores the coordinates, time, type and other parameters of the operation;

(6) After the operation is over, the "Bike" APP runs successfully, and the information acquisition module saves another frame of the image after the operation;

(7) Repeat steps (3) to (6) until the user ends the recording;

(8) The control module generates a shortcut.

2. The shortcut execution process includes the following steps:

(1) An entry will be added on the interface for users to start shortcuts, or users can set the automatic startup time of shortcuts. When the shortcuts need to be executed, the control module obtains the operation with the serial number of 0001 from the storage module, and takes Various parameters and keyframe images to the operation;

(2) Collect the current screenshot image through the information acquisition module, compare the current image with the key frame before the operation of No. 0001, and determine whether the “Bike” APP icon is still still present through image recognition technology (if there is a “Danche” on the interface. ” APP icon, or “Bicycle” APP icon, or “Bike” APP icon, it is also considered that the icon corresponding to the “Bicycle” APP is still there), whether the location has changed:

If there is no change, perform operation 0001;

If there is a change in the position of the "Bike" icon, it needs to be corrected. Through the image recognition technology, the new coordinates of the "bicycle" application icon can be identified, and then the control module stores the new coordinates in the corresponding place of the storage module, and performs operation 0001 according to the new coordinates;

If the "Bike" icon is gone, the control module prompts the user that the operation fails;

(3) After the 0001 operation is completed, a frame of image is collected, and compared with the key frame after the 0001 operation stored in advance to determine whether the operation is successful:

If there is no change, it means the execution is successful; you can continue to the next step;

If the "Bike" application fails to start, it will prompt the user that the operation failed, and the process will be terminated abnormally;

(4) If step (3) is successfully executed, repeat steps (1) to (3) for the operation whose serial number is 0002; until all operations are completed, or the process terminates abnormally.

3. Error correction process:

The reason for the need for error correction may be due to the change of the position of the "Bicycle" APP on the desktop. Through real-time image recognition technology, it is possible to identify which area in the current screenshot is the icon of the "bicycle" APP, or identify (or further identify) the two Chinese characters "bicycle", and then obtain the "bicycle" APP in the screenshot. Location. And the new position of the "Bicycle" APP in the screenshot is the new coordinates to perform the screen click operation.

For example, Figure 6 is a schematic diagram of the movement of the application coordinates. As shown in Figure 6, the left picture is the position of the "bicycle" coordinates when the user records the shortcut, and then the user changes the coordinates of the "bicycle" application for some reason. , changed to the position shown on the right in Figure 6. Assuming that the coordinates of the upper left corner of the screenshot are (0, 0), through image recognition technology, it can be obtained that the position of the "bicycle" application in the left picture in Figure 6 is (x1, y1), and the position in the right picture is (x2 , y2). Since the icon has a certain width and height, when taking the coordinates, it is recommended to take the coordinates of the center point of the icon.

Obtaining the coordinates in the right diagram from the coordinates in the left diagram in FIG. 6 can be implemented in the following manner.

In general, the coordinates in the screenshot and the coordinates of the click on the screen are in one-to-one correspondence, that is to say, the coordinates of the "bicycle" icon on the screenshot are the coordinates of the user's click on the screen. In this case (x2, y2) is the new click coordinate of "bicycle";

A special case is that the coordinates in the screenshot are not the coordinates of the click on the screen. Suppose, the coordinates of "bicycle" in the left picture of Figure 6 are (x1, y1), the coordinates of the user's click are (xd1, yd1), and the coordinates of "bicycle" in the right picture of Figure 6 are (x2, y2), since the coordinates in the screenshot and the coordinates of the clicked screen have a corresponding relationship, (xd2, yd2) can be calculated by the following algorithm:

(x1,y1)/(xd1,yd1)=(x2,y2)/(xd2,yd2)

Among them, (x1, y1), (x2, y2) are the coordinates obtained by the image recognition technology in the screenshot, and (xd1, yd1) are the coordinates of the user's click on the screen collected by the information collection module when the user is recording.

Fourth, the judgment process of the operation result:

According to the previously recorded image of the interface of the user's normal operation to open the "Bicycle" application, whether the image after the current operation is consistent with the image can be determined by comparing the images. Relying on image recognition technology, it is easy to judge whether the "bicycle" application is running successfully.

Not only the operation of starting the application, but also the formation of other types of operations can also be judged through image recognition technology. For example, the state of the switch can also be judged through image recognition technology, and it is possible to know whether the current state of the switch is on or off. FIG. 7 is a schematic diagram of different switch states. As shown in FIG. 7 , the image on the left is an image with the switch off, and the image on the right is an image with the switch on. The difference between the two is obvious.

In short, with the use of image recognition technology, the system seems to have a pair of "eyes", which can accurately identify whether the preconditions for each operation are met, and whether the results of the operation are in line with expectations.

To sum up, the embodiments of the present disclosure allow users to customize shortcuts, and since the image recognition technology is combined, errors can be corrected during the execution of the shortcuts, and the execution result can be judged.

Obviously, those skilled in the art should understand that the above-mentioned modules or steps of the present disclosure can be implemented by a general-purpose computing device, and they can be centralized on a single computing device or distributed in a network composed of multiple computing devices On the other hand, they can be implemented in program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, can be performed in a different order than shown here. Or the described steps, or they are respectively made into individual integrated circuit modules, or a plurality of modules or steps in them are made into a single integrated circuit module to realize. As such, the present disclosure is not limited to any particular combination of hardware and software.

The above descriptions are only the embodiments and exemplary embodiments of the present disclosure, and are not intended to limit the present disclosure. For those skilled in the art, the present disclosure may have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the principles of the present disclosure shall be included within the protection scope of the present disclosure.

Claims

A method for obtaining an operation collection, including:

Receive one or more operations on the terminal device, and obtain operation information of each operation in the one or more operations, wherein the operation information includes: an operation information used to identify the operation in the one or more operations The sequence identification information of the operation sequence, the operation description data of the operation;

An operation set is generated according to the operation information, wherein the operation set includes: the operation information of the one or more operations.
The method according to claim 1, wherein the operation information further comprises: a relevant frame image corresponding to the operation, wherein the relevant frame image comprises: a valid frame image before execution, a valid frame image at the time of execution, and a valid frame image after execution Valid frame image.
The method according to claim 2, wherein receiving one or more operations on the terminal device and acquiring operation information of each of the one or more operations comprises:

In response to the received operation set collection request, the terminal device obtains the relevant frame image corresponding to each operation in the one or more operations through a screen recording function or a screen capture function, and collects the one or more operations The sequence identification information and the operation description data of each operation in the . until a collection end indication is received.
The method of claim 1, wherein,

The sequence identification information used to identify the operation sequence of the operation in the one or more operations includes at least one of the following: the operation time of the operation, the operation sequence number of the operation in the one or more operations;

and / or,

The operation description data of the operation includes at least one of the following: operation category, coordinate parameter, duration parameter, key identification information, identification information of the sensor that collects biometrics, acquisition parameters for collecting biometrics, and operation object corresponding to the operation , execution page description information, and result page description information.
The method of claim 4, wherein,

The operation category is obtained based on a screen touch signal or a button touch signal or a system sensor call signal; and/or,

The coordinate parameters are obtained based on the screen touch signal; and/or,

The duration parameter is obtained based on the screen touch signal; and/or,

The key identification information is obtained based on a key touch signal; and/or,

The identification information and the acquisition parameters of the sensors that collect biological features are acquired based on the system sensor call signal; and/or,

The operation object corresponding to the operation and the description information of the execution page are obtained based on the image recognition technology according to the valid frame image corresponding to the execution time of the operation, or based on the image recognition technology based on the valid frame image before the execution corresponding to the operation combined with the coordinate parameters obtain; and/or,

The description information of the result page corresponding to the operation is obtained based on the image recognition technology according to the effective frame image after execution corresponding to the operation.
The method according to any one of claims 1-5, wherein after generating the operation set according to the operation information, it further comprises at least one of the following:

save the set of operations;

After setting the execution condition corresponding to the operation set, save the operation set and the execution condition corresponding to the operation set;

sending the set of operations;

After setting the execution condition corresponding to the operation set, the operation set and the execution condition corresponding to the operation set are sent.
An execution method of an operation collection, including:

In the case of receiving an operation set execution request corresponding to the operation set or judging that the execution condition corresponding to the operation set is satisfied, the operation set is obtained, wherein the operation set includes: operation information of one or more operations, the operation The information includes: sequence identification information for identifying the operation sequence of the operation in the one or more operations, and operation description data of the operation;

The one or more operations are performed according to the operation description data according to the operation sequence identified by the sequence identification information.
The method according to claim 7, wherein the operation information further comprises: a relevant frame image corresponding to the operation, wherein the relevant frame image comprises: a valid frame image before execution, a valid frame image at the time of execution, and a valid frame image after execution Valid frame image.
The method according to claim 8, wherein, according to the operation sequence identified by the sequence identification information, performing the one or more operations according to the operation description data comprises:

Determine the current operation to be performed according to the sequence identification information;

Determine whether the precondition for executing the current operation is satisfied according to the current screen image before execution and the valid frame image before execution corresponding to the current operation, and execute the current operation if it is satisfied;

It is determined whether the current operation is successfully executed, and if the execution is successful, the next current operation to be executed is continued to be determined and executed until the one or more operations are executed.
The method according to claim 9, wherein whether the precondition of the current operation is satisfied is determined according to the current pre-execution screen image and the pre-execution valid frame image corresponding to the current operation, and if the precondition is satisfied Performing the current operation below includes:

determining whether the current screen image before execution includes the operation object corresponding to the current operation;

If the current pre-execution screen image includes an operation object corresponding to the current operation, in the case where the position of the operation object corresponding to the current operation is the same in the current pre-execution screen image and the pre-execution effective frame image In the next step, the current operation is performed according to the operation description data of the current operation; and/or, in the screen image before the current execution and the effective frame image before the execution of the operation object corresponding to the current operation When the position changes, adjust the operation description data of the current operation according to the position of the operation object corresponding to the current operation in the screen image before the current operation, and adjust the operation description data of the current operation according to the adjusted value of the current operation. The operation description data performs the current operation.
The method according to claim 10, wherein determining whether the current pre-execution screen image includes an operation object corresponding to the current operation comprises at least one of the following:

Based on image recognition technology, identify whether to include the icon of the operation object corresponding to the current operation in the screen image before the current execution, and determine whether the screen image before the current execution includes the operation object corresponding to the current operation according to the recognition result;

Identify the page description information included in the current pre-execution screen image based on image recognition technology, match the identified page description information with the execution page description information corresponding to the current operation, and determine the current pre-execution page description information according to the matching result. Whether the screen image includes the operation object corresponding to the current operation.
The method according to claim 10, wherein adjusting the operation description data of the current operation according to the position of the operation object corresponding to the current operation in the screen image before the current execution comprises one of the following:

In the case that the coordinate parameters of the operation object in the screen image before the current execution are consistent with the coordinate parameters of the click on the operation object on the screen, the coordinate parameters in the operation description data of the current operation are changed to Replaced with the coordinate parameters of the operation object corresponding to the current operation in the screen image before the current execution;

In the case that the coordinate parameters of the operation object in the screen image before the current execution are inconsistent with the coordinate parameters of clicking the operation object on the screen, the coordinates of the operation object after the click changes position are determined according to the following formula Parameter (xd2, yd2): (x1, y1)/(xd1, yd1)=(x2, y2)/(xd2, yd2), where (x1, y1) is the valid frame of the operation object before the execution The coordinate parameters in the image, (x2, y2) are the coordinate parameters of the operation object in the screen image before the current execution, and (xd1, yd1) are the coordinate parameters in the operation description data before adjustment.
The method according to claim 10, wherein, in the case that the current pre-execution screen image does not include an operation object corresponding to the current operation, the method further comprises one of the following:

confirming the failure to perform the one or more operations;

Return to repeat the previous operation of the current operation;

prompting the user to proceed with the unfinished operation of the one or more operations;

Prompt the user to perform the current operation, and after the current operation is completed, continue to follow the sequence of operations identified by the sequence identification information, and execute the unfinished operations in the one or more operations according to the operation description data .
The method of claim 9, wherein determining whether the current operation is successfully performed comprises at least one of the following:

Identify whether the screen image after the current execution is consistent with the effective frame image after execution corresponding to the current operation based on image recognition technology, and determine that the current operation is successfully executed if they are consistent;

Identify, based on image recognition technology, whether the current screen image includes an icon of an operation object corresponding to the next operation of the current operation, and if it is included, determine that the current operation is successfully executed;

Identify the page description information included in the currently executed screen image based on the image recognition technology, determine whether the identified page description information matches the result page description information corresponding to the current operation, and determine the current The operation performed successfully.
The method according to any one of claims 7-14, wherein before acquiring the operation set, it further comprises one of the following:

Receive the one or more operations on the terminal device, obtain operation information of each operation in the one or more operations, generate the operation set according to the operation information, and save the operation set;

receiving the operation set sent by other terminal equipment;

Receive the one or more operations on the terminal device, obtain operation information of each operation in the one or more operations, generate the operation set according to the operation information, and set the corresponding execution conditions, and save the operation set and the execution conditions corresponding to the operation set;

The operation set and the execution condition corresponding to the operation set sent by other terminal devices are received.
A computer-readable storage medium in which a computer program is stored, wherein the computer program is configured to execute the method according to any one of claims 1 to 6 when running, or A method as claimed in any one of claims 7-15 is performed.
A terminal device, comprising a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the method according to any one of claims 1 to 6, Or perform the method of any one of claims 7-15.