CN112578979B - Method and device for generating operation script, readable medium and equipment - Google Patents
Method and device for generating operation script, readable medium and equipment Download PDFInfo
- Publication number
- CN112578979B CN112578979B CN202011580232.0A CN202011580232A CN112578979B CN 112578979 B CN112578979 B CN 112578979B CN 202011580232 A CN202011580232 A CN 202011580232A CN 112578979 B CN112578979 B CN 112578979B
- Authority
- CN
- China
- Prior art keywords
- control object
- information
- operation intention
- intention information
- target
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
- G06F3/0487—Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
- G06F3/0481—Interaction techniques based on graphical user interfaces [GUI] based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/451—Execution arrangements for user interfaces
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02P—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS
- Y02P90/00—Enabling technologies with a potential contribution to greenhouse gas [GHG] emissions mitigation
- Y02P90/02—Total factory control, e.g. smart factories, flexible manufacturing systems [FMS] or integrated manufacturing systems [IMS]
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Human Computer Interaction (AREA)
- Biomedical Technology (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- Health & Medical Sciences (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- Mathematical Physics (AREA)
- User Interface Of Digital Computer (AREA)
Abstract
The application discloses a method, a device, a readable medium and equipment for generating an operation script, wherein the method comprises the steps of collecting an interface image and a corresponding timestamp in an operation flow, and collecting operation information in the operation flow; inputting the operation information into an operation intention recognition model, and recognizing to obtain operation intention information of each control object to be recognized; for each control object to be identified, matching the control object to be identified with the operation intention information of the control object to be identified to obtain a target control object from each control object in each interface image; associating the target control object with the operation intention information of the control object to be identified to obtain operation intention information corresponding to the target control object; and generating an operation control script corresponding to the operation flow by using the operation intention information corresponding to each target control object. Because the operation control script obtained by the application is executed according to the operation intention, the reliability is higher compared with the prior art for realizing the operation purpose.
Description
Technical Field
The present application relates to the field of computer technologies, and in particular, to a method and an apparatus for generating an operation script, a readable medium, and a device.
Background
In the prior art, repetitive operation work is usually written into a corresponding operation script, and then the operation script is automatically and repeatedly executed to replace manual repeated operation. The existing method for generating the operation script mainly records the mouse operation action and the coordinates in the manual operation process to further generate the operation script. When the operation script is executed, the operation action and the movement track of the mouse can be repeated, and the purpose of automatically executing the operation is achieved.
However, the operation script obtained by the existing method for generating the operation script is not necessarily capable of successfully repeating the manual operation process. For example, the operations performed by the user are: and operating the mouse wheel, scrolling for five times, moving to a certain control, and double-clicking the control. However, in the process of automatically executing the operation script by the actual machine, the scroll distance generated when the machine executes five times of scrolling operations is not necessarily consistent with the scroll distance generated in the actual user operation, so that the position to which the mouse is finally moved is not necessarily on the control, the control is not aligned for execution when the double-click operation is executed, and the operation purpose of clicking the control is not achieved. The operation script generated in the prior art cannot accurately restore the flow of manual operation, cannot be executed to achieve the operation purpose, and has low execution reliability.
Disclosure of Invention
Based on the defects of the prior art, the application provides a method, a device, a readable medium and a device for generating an operation script, so as to improve the reliability of the execution of the operation control script.
The first aspect of the present application discloses a method for generating an operation script, including:
in the process that a target user executes an operation flow, acquiring an interface image and a corresponding timestamp in the operation flow, and acquiring operation information of the target user in the operation flow;
inputting the operation information of the target user into an operation intention recognition model, and recognizing to obtain the operation intention information of the target user on each control object to be recognized in the operation flow; the operation intention recognition model is obtained by training a neural network model through operation information of a plurality of users and actual operation intention information of each user; the operation intention information of each control object to be identified is used for explaining the operation purpose of the target user on the control object to be identified;
for each control object to be identified, matching and obtaining a target control object corresponding to the control object to be identified from each control object in each interface image by using the operation intention information of the control object to be identified;
associating the target control object with the operation intention information of the control object to be identified to obtain operation intention information corresponding to the target control object;
generating an operation control script corresponding to the operation flow by using the operation intention information corresponding to each target control object; the operation control script is used for identifying each target control object positioned in an equipment interface by respectively utilizing the operation intention information corresponding to each target control object according to the time stamp sequence of the operation intention information corresponding to each target control object, and executing the corresponding operation intention on each target control object.
Optionally, in the above method for generating an operation script, the operation intention information of the control object to be identified includes: the target user performs an operation intention action on the control to be recognized, executes a timestamp corresponding to the operation intention action, and executes a coordinate corresponding to the operation intention action;
the step of obtaining, by matching, for each control object to be recognized, a target control object corresponding to the control object to be recognized from each control object in each interface image by using the operation intention information of the control object to be recognized includes:
for each control object to be identified, determining an interface image matched with a timestamp in the operation intention information from each interface image by using the timestamp in the operation intention information of the control object to be identified;
and determining a target control object matched with the coordinates in the operation intention information from each control object of the matched interface image by using the coordinates in the operation intention information of the control object to be identified.
Optionally, in the above method for generating an operation script, the method for creating an operation intention recognition model includes:
constructing a training data set; wherein the training data set comprises: operation information of a plurality of users and actual operation intention information of each of the users; the operation information comprises: the method comprises the following steps of (1) carrying out a plurality of operation actions, a timestamp corresponding to each operation action and a coordinate corresponding to each operation action;
respectively inputting the operation information of each user in the training data set into a neural network model, and respectively predicting to obtain the operation intention information of each user;
continuously adjusting the weight and the threshold value in the neural network model according to the predicted error between the operation intention information of each user and the corresponding actual operation intention information until the error between the operation intention information of each user and the corresponding actual operation information obtained by the adjusted neural network model meets a preset convergence condition, and determining the adjusted neural network model as an operation intention identification model.
Optionally, in the method for generating an operation script, before determining, from each control object of the matched interface image, a target control object matched with the coordinates in the operation intention information by using the coordinates in the operation intention information of the control object to be identified, the method further includes:
and for each interface image, carrying out control object identification on the interface image, and identifying to obtain each control object and a control object corresponding coordinate in the interface image.
Optionally, in the above method for generating an operation script, the operation control script includes: the operation intention information corresponding to each target control object and the identification algorithm adopted for identifying each target control object.
Optionally, in the method for generating an operation script, the acquiring an interface image and a corresponding timestamp in the operation flow includes:
and triggering the interface to capture the screen when the target user executes the operation action each time, so as to obtain each interface image and the corresponding timestamp.
A second aspect of the present application discloses an apparatus for generating an operation script, including:
the acquisition unit is used for acquiring an interface image and a corresponding timestamp in an operation flow and acquiring operation information of a target user in the operation flow in the process of executing the operation flow by the target user;
the intention identification unit is used for inputting the operation information of the target user into an operation intention identification model, and identifying and obtaining the operation intention information of the target user to each control object to be identified in the operation flow; the operation intention recognition model is obtained by training a neural network model through operation information of a plurality of users and actual operation intention information of each user; the operation intention information of each control object to be identified is used for explaining the operation purpose of the target user on the control object to be identified;
the matching unit is used for matching each control object to be identified to obtain a target control object corresponding to the control object to be identified from each control object in each interface image by using the operation intention information of the control object to be identified;
the association unit is used for associating the target control object with the operation intention information of the control object to be identified to obtain the operation intention information corresponding to the target control object;
the generating unit is used for generating an operation control script corresponding to the operation flow by using the operation intention information corresponding to each target control object; the operation control script is used for identifying each target control object positioned in an equipment interface by respectively utilizing the operation intention information corresponding to each target control object according to the time stamp sequence of the operation intention information corresponding to each target control object, and executing the corresponding operation intention on each target control object.
Optionally, in the above apparatus for generating an operation script, the operation intention information of the control object to be identified includes: the target user performs an operation intention action on the control to be recognized, executes a timestamp corresponding to the operation intention action, and executes a coordinate corresponding to the operation intention action;
wherein the matching unit includes:
the first determining subunit is configured to determine, for each control object to be identified, an interface image that matches the timestamp in the operation intention information from each interface image by using the timestamp in the operation intention information of the control object to be identified;
and the second determining subunit is used for determining a target control object matched with the coordinates in the operation intention information from each control object in the matched interface image by using the coordinates in the operation intention information of the control object to be identified.
Optionally, in the apparatus for generating an operation script, the apparatus further comprises:
the building unit is used for building a training data set; wherein the training data set comprises: operation information of a plurality of users and actual operation intention information of each of the users; the operation information comprises: the method comprises the following steps of (1) carrying out a plurality of operation actions, a timestamp corresponding to each operation action and a coordinate corresponding to each operation action;
the input unit is used for respectively inputting the operation information of each user in the training data set into a neural network model and respectively predicting to obtain the operation intention information of each user;
and the adjusting unit is used for continuously adjusting the weight and the threshold value in the neural network model according to the error between the predicted operation intention information of each user and the corresponding actual operation intention information until the error between the operation intention information of each user and the corresponding actual operation information obtained by the adjusted neural network model meets a preset convergence condition, and determining the adjusted neural network model as an operation intention identification model.
Optionally, the operation script generating device further includes:
and the identification unit is used for identifying the control object of each interface image and identifying each control object and the corresponding coordinate of the control object in the interface image.
Optionally, in the above apparatus for generating an operation script, the operation control script includes: the operation intention information corresponding to each target control object and the identification algorithm adopted for identifying each target control object.
Optionally, in the above apparatus for generating an operation script, the acquiring unit includes:
and the triggering unit is used for triggering the screenshot of the interface when the target user executes the operation action each time to obtain each interface image and the corresponding timestamp.
A third aspect of the application discloses a computer readable medium having a computer program stored thereon, wherein the program when executed by a processor implements the method as described in any of the first aspects above.
The present application fourth aspect discloses an apparatus comprising:
one or more processors;
a storage device having one or more programs stored thereon;
the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the method as in any one of the first aspects above.
According to the technical scheme, in the method for generating the operation script, the operation information of the target user is input into the operation intention identification model, and the operation intention information of the target user for each control object to be identified in the operation process is identified and obtained. The operation intention identification model is obtained by training a neural network model through operation information of a plurality of users and actual operation intention information of each user, the operation intention information of each to-be-identified control object is used for explaining the operation purpose of a target user to the to-be-identified control object, the operation intention information of the to-be-identified control object is utilized for each to-be-identified control object, a target control object corresponding to the to-be-identified control object is obtained through matching from each control object in each interface image, then the target control object is associated with the operation intention information of the to-be-identified control object, the operation intention information corresponding to the target control object is obtained, an operation control script corresponding to an operation flow is generated through the operation intention information corresponding to each target control object, and therefore the operation control script can identify each target control object positioned in an equipment interface according to the time stamp sequence of the operation intention information corresponding to each target control object through the operation intention information corresponding to each target control object, and corresponding operation intention is executed on each target control object. Because the operation control script obtained in the embodiment of the application is executed according to the operation intention, compared with a control script which only repeats operation actions and movement tracks in the prior art, the operation control script in the embodiment of the application has higher reliability in achieving the operation purpose, and can accurately restore the flow of manual operation.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the provided drawings without creative efforts.
Fig. 1 is a schematic flowchart of a method for generating an operation script according to an embodiment of the present application;
fig. 2 is a schematic flowchart of a method for constructing an operation intention recognition model according to an embodiment of the present application;
fig. 3 is a schematic flowchart of a method for obtaining a target control object through matching according to an embodiment of the present application;
fig. 4 is a schematic structural diagram of an operation script generating apparatus according to an embodiment of the present application.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be obtained by a person skilled in the art without making any creative effort based on the embodiments in the present invention, belong to the protection scope of the present invention.
Referring to fig. 1, the embodiment of the present application discloses a method for generating an operation script, which specifically includes the following steps:
s101, in the process that a target user executes an operation flow, acquiring an interface image and a corresponding timestamp in the operation flow, and acquiring operation information of the target user in the operation flow.
When a target user wants to generate an operation control script to repeatedly execute a certain operation process, the operation process required to be repeatedly executed by the target user is collected. Specifically, in the process of executing the operation flow by the target user, an interface of the target user during the operation flow and operation information of the target user in the operation flow are collected. The operation information of the target user refers to information for describing an operation performed by the target user in the operation flow. Such as the operation action of the target user (mouse click, keyboard input, scroll wheel, slide, etc.), the time when the operation action occurs (i.e., the timestamp corresponding to the operation action), the coordinates corresponding to the operation action, the distance the mouse moves, etc. The more detailed the operation information, the higher the execution reliability of the finally obtained operation control script.
The interface image acquisition mode may specifically be recording an interface in an operation flow, and then converting the recorded operation interface video into interface images in a plurality of operation flows and corresponding timestamps. Or screenshot can be performed on the interface images in the operation flow according to a preset acquisition period, and the screenshot time corresponding to each interface image is stored. And triggering the screenshot of the interface when the target user executes the operation action each time to obtain each interface image and the corresponding timestamp. For example, a screen capture is triggered when a mouse click action is performed, and a screen capture is performed when a keyboard input operation is performed. Therefore, the screenshot times are saved, and the screenshot of the key change of the interface image caused by external input is selectively recorded.
S102, inputting the operation information of the target user into an operation intention recognition model, and recognizing to obtain the operation intention information of the target user on each control object to be recognized in the operation process, wherein the operation intention recognition model is obtained by training a neural network model through the operation information of a plurality of users and the actual operation intention information of each user, and the operation intention information of each control object to be recognized is used for explaining the operation purpose of the target user on the control object to be recognized.
All the operation information of the target user in the operation flow obtained in the step S101 is input into the operation intention recognition model, and the operation intention information of the target user for each control object to be recognized in the operation flow is recognized by the operation intention recognition model. The control object to be recognized refers to a control object which can be operated in the unrecognized interface. The operation intention information of the control object to be identified is used for explaining the operation purpose of the target user on the control object to be identified. In the process of executing the operation flow, a user often additionally executes a plurality of operations for achieving the purpose of operating the control object. For example, the operation information includes 9 times of mouse wheel, the mouse moves to a certain coordinate to perform double-click operation, the purpose of the series of operations is to click the control object to be recognized at a certain coordinate position through the operation of mouse wheel sliding and moving, and what control object is specifically unknown at the coordinate position.
The operation intention recognition model is obtained by training the neural network model through the operation information of a plurality of users and the actual operation intention information of each user, so that the operation intention recognition model can recognize the operation intention information of each control to be recognized in the operation flow through the operation information.
For example, if the operation information records that the mouse moves to the coordinate a and then clicks, the keyboard inputs a plurality of data, then moves to the coordinate B again and then clicks. After the series of operation information is input to the operation intention recognition model, it is known that the operation intention corresponding to the operation information is: the operation purposes of clicking and inputting a plurality of data are completed by moving the control object to be identified to the position of the control object to be identified of the coordinate A, and the clicking operation is completed by moving the control object to be identified to the position of the control object to be identified of the coordinate B.
In the prior art, the operation script is generated mainly by recording mouse operation actions and coordinates in a manual operation process. When the operation script is executed, the operation action and the movement track of the mouse can be repeated, and the purpose of automatically executing the operation is achieved. For example, the operations performed by the user are: operating the mouse wheel, scrolling for five times, moving to a certain coordinate position, and then double clicking. The operation script finally generated by the existing mode is low in reliability, and the machine cannot completely restore the operation same as that of a user, for example, when the user's scroll operation is repeated, the moving distance generated by the machine controlled scroll may be inconsistent with the moving distance generated by the user through the mouse roller scroll, so that the user does not really move to the position of the control to be clicked when finally moving to a certain coordinate position. The operation script is mechanized to repeat the operation process of the user, which finally causes that the operation purpose in the operation process is difficult to realize and the operation process of the user cannot be really repeated.
In the embodiment of the application, the operation intention recognition model is obtained by training the neural network model through the operation information of a plurality of users and the actual operation intention information of each user, so that the operation intention recognition of the operation information of the target user can be performed through the operation intention recognition model, the operation intention information of the target user on each control object to be recognized in the operation process, namely the operation purpose of the target user on each control object to be recognized is obtained, the operation control script is converted into a script capable of achieving the operation purpose of each control object, and the purpose of improving the reliability of the operation control script is achieved. For example, the operations performed by the user are: operating the mouse wheel, scrolling for five times, moving to a certain coordinate position, and then double-clicking. The operation intention recognition model is recognized to find a control object to be recognized at a certain coordinate position through the sliding and moving operation of the roller, and the purpose of double-clicking the control execution object to be recognized is achieved. The operation intention information of each control to be recognized output by the operation intention recognition model does not pay attention to the detail information of repeated operations (such as the number of times of repeating the roller and the number of sliding) executed in the process of achieving the operation purpose, but mainly pays attention to the final purpose of a series of operations, such as the operation purpose of clicking one control object to be recognized, and then an operation control script is generated in a mode of restoring the operation intention of a target user, so that the execution reliability of the operation control script is improved.
Optionally, in a specific embodiment of the present application, the operation intention information of the control object to be recognized includes: the target user executes the operation intention action of the control to be recognized, the timestamp corresponding to the operation intention action and the coordinate corresponding to the operation intention action.
The operation intention action refers to an action for achieving the operation purpose. For example, when the scroll wheel slides to a certain control object to be recognized and then the mouse clicks the control object to be recognized, the operation intention action is the mouse click. And the time stamp corresponding to the action of executing the operation intention is the time stamp clicked by the mouse, and the coordinate corresponding to the action of executing the operation intention is the coordinate position clicked by the mouse. Optionally, actions in the process of realizing the operation intention can be included in the operation intention information. For example, the purpose of clicking the control to be recognized is achieved through a roller and a moving mouse, and then the roller and the moving mouse belong to actions in the process of realizing the operation intention.
Optionally, referring to fig. 2, in a specific embodiment of the present application, the process of constructing the operation intention recognition model includes:
s201, constructing a training data set, wherein the training data set comprises: the operation information of a plurality of users and the actual operation intention information of each user, the operation information comprises: the system comprises a plurality of operation actions, a timestamp corresponding to each operation action and coordinates corresponding to each operation action.
Specifically, the operation information of the user in a plurality of operation flows is collected in advance, and then the operation information of the user in each flow and the corresponding actual operation intention information are correspondingly associated and stored in a training data set. Wherein the more information in the training data set, the higher the accuracy of the trained operation intention recognition model.
Optionally, for each operation flow, operation information executed by a plurality of users for the operation flow may be collected, and the operation information executed by each user for the operation flow is corresponding to the same actual operation intention information, and is correspondingly stored in the training data set.
S202, respectively inputting the operation information of each user in the training data set into the neural network model, and respectively predicting to obtain the operation intention information of each user.
The neural network model may be a sequence-to-sequence model, a convolutional neural network model, or the like. After the operation information of each user in the training data set is respectively input into the neural network model, the operation intention information corresponding to the operation information of each user, namely the corresponding operation intention information for each control object to be identified, is respectively predicted by the neural network model.
And S203, continuously adjusting the weight and the threshold value in the neural network model according to the predicted error between the operation intention information of each user and the corresponding actual operation intention information until the error between the operation intention information of each user and the corresponding actual operation information, which are obtained by the adjusted neural network model, meets a preset convergence condition, and determining the adjusted neural network model as an operation intention identification model.
And continuously adjusting the weight and the threshold value in the neural network model to make the error between the operation intention information of each user and the corresponding actual operation intention information smaller and smaller until the error between the operation intention information of each user and the corresponding actual operation information obtained by the adjusted neural network model meets a preset convergence condition, and then considering that the adjusted neural network model can accurately predict intention identification information, and further determining the adjusted neural network model as the operation intention identification model.
S103, aiming at each control object to be identified, matching each control object in each interface image by using the operation intention information of the control object to be identified to obtain a target control object corresponding to the control object to be identified.
Since only the operation information is input to the operation intention recognition model in step S102, it cannot be determined which control objects are operated, and further, the control objects to be recognized related in step S102 need to be matched and recognized through the interface image. Specifically, each control object in the interface image is identified in advance, and then a target control object corresponding to the control object to be identified is obtained by matching from each control object in each interface image by using the operation intention information of the control object to be identified.
Optionally, in a specific embodiment of the present application, before executing step S103, the method further includes:
and for each interface image, carrying out control object identification on the interface image, and identifying to obtain each control object and the corresponding coordinate of the control object in the interface image.
Specifically, the process of identifying the control object in advance for each interface image may be: and for each interface image, dividing the interface image into a plurality of areas, for the image of each area, matching the image of the area with the image of each known control object in a prestored known control object gallery, taking the matched known control object as the control object of the image of the area, and correspondingly storing the coordinates of the control object and the name of the identified control object. For example, the degree of similarity between the feature points in the image of the region and the feature points in the image of each known control object may be compared, the degree of similarity between the image of the region and the image of each known control object may also be compared, and finally, the known control object with the similarity higher than the threshold value is used as the control object in the image of the region. And for the control object which is not identified from the control object gallery, the unknown control object in the interface image can be directly intercepted and named, and the named name and the coordinate of the unknown control object are correspondingly stored. Or obtaining the feature information of the unknown control object in the image interface by using some algorithms, naming the unknown control object, and storing the feature information of the control object, the used algorithm for extracting the feature information, the named name and the corresponding coordinates in an associated manner. Alternatively, there are many algorithms for extracting feature information of an unknown control object, for example, a separation algorithm may be used to obtain feature points of the unknown control object, a corner detection algorithm may be used to obtain feature points of the unknown control object, a character recognition algorithm may be used to obtain a position of the unknown control object where a specific character is located, and so on.
Optionally, referring to fig. 3, if the operation intention information of the control object to be recognized includes: if the target user performs an operation intention action on the control to be recognized, a timestamp corresponding to the execution operation intention action, and a coordinate corresponding to the execution operation intention action, an implementation manner of step S103 is executed, which includes:
s301, aiming at each control object to be identified, determining an interface image matched with the time stamp in the operation intention information from each interface image by using the time stamp in the operation intention information of the control object to be identified.
Since the time stamp of each interface image is also acquired and recorded when the interface image is acquired in step S101, an interface image (i.e., an interface image matched with the time stamp in the operation intention information) consistent with the time stamp in the operation intention information can be found by using the time stamp in the operation intention information of the control object to be recognized, and an interface displayed by the interface image is an interface image when the target user performs an operation intention action on the control to be recognized.
S302, determining a target control object matched with the coordinates in the operation intention information from each control object of the matched interface image by using the coordinates in the operation intention information of the control object to be identified.
The interface displayed by the interface image matched with the timestamp in the operation intention information obtained in step S302 is consistent with the timestamp of the target user when the target user performs the operation intention action on the control to be recognized, so that the control to be recognized exists in the interface image. And the operation intention information also comprises coordinates corresponding to the action of executing the operation intention, so that the position of the coordinates corresponding to the action of executing the operation intention is the position of the control object to be recognized. And since each control object in the matched interface image is identified in advance, only the coordinates in the operation intention information of the control object to be identified need to be matched with the coordinates corresponding to each control object in the interface image, and the control object closest to the coordinates in the operation intention information of the control object to be identified in the interface image is taken as the target control object matched with the coordinates in the operation intention information. The target control object is the control object to be identified, and the identification of the control object to be identified is completed.
For example, the operation intention information of a certain control object to be identified includes a click operation of a target user on the control object to be identified, the click operation occurs at a time point a, and then an interface image captured at the time point a is found from each interface image, where the interface image includes a control object a, a control object C, and a control object B. And the coordinate included in the operation intention information of the control object to be identified is closest to the coordinate of the control object C, so that the control object C is the target control object, namely the control object to be identified is identified to be the control object C.
And S104, associating the target control object with the operation intention information of the control object to be identified to obtain the operation intention information corresponding to the target control object.
Since the target control object matched with the coordinates in the operation intention information of the control object to be recognized is determined for each control object to be recognized in step S103, it can be considered that the control object to be recognized is the target control object, and therefore the operation intention information of the control object to be recognized is the operation intention information of the target control object (i.e., the operation intention information corresponding to the target control object).
And for each control object to be identified, associating the target control object determined by the intention identification information of the control object to be identified with the operation intention information of the control object to be identified, and finally obtaining the operation intention information corresponding to the target control object. The operation intention information of the target control object may include related information of the target control object, for example, coordinate information of the target control object in the interface image, name information of the target control object, and operation intention information such as an operation intention action for the target control object and a timestamp corresponding to the operation intention action.
And S105, generating an operation control script corresponding to the operation flow by using the operation intention information corresponding to each target control object, wherein the operation control script is used for identifying each target control object positioned in the equipment interface by using the operation intention information corresponding to each target control object according to the time stamp sequence of the operation intention information corresponding to each target control object, and executing the corresponding operation intention on each target control object.
And writing the operation intention information corresponding to each target control object into the operation control script according to the time stamp sequence by using the operation intention information corresponding to each target control object, so that the operation control script can respectively complete the operations of identifying, positioning and executing the operation intention on each target control object according to the time stamp sequence.
The writing language of the operation control script can be a natural language or other types of programming languages, and is specifically determined according to actual requirements.
For example, the operation intention information of each target control object is arranged according to the time stamp sequence, and the sequence is as follows: the method comprises the steps of firstly moving the target control object A to execute the operation intention action of mouse clicking, then moving the target control object B through the scrolling operation, and executing the double-click operation. The control script generated may be either: the method comprises the steps of firstly identifying a target control object A in an interface image, then moving a mouse to the position of the target control object A, executing mouse clicking operation, then executing scrolling operation, moving to the position of the target control object B until the target control object B is identified and located in the interface, and executing double-click operation.
Compared with the prior art, the operation control script is in a mode of repeating operation actions and moving tracks in the operation control script through machinery, the operation flow is converted into the operation intention of each target control object, so that the operation control script can be executed in a mode of finishing the operation intention of each target control object, and the execution reliability of the operation control script is improved.
Optionally, in a specific embodiment of the present application, the operating the control script includes: the operation intention information corresponding to each target control object and the identification algorithm adopted for identifying each target control object.
In other words, the operation control script writes in the operation intention information corresponding to each target control object and also writes in an identification algorithm adopted when identifying the target control object, so as to accurately position the target control object in the interface. There are many recognition algorithms, for example, the target control object in the interface image can be found by matching the image template of the target control object with the interface image, so as to complete positioning. And calculating the characteristic points of each control object in the interface image through a corner detection algorithm, finding out the position of the control object matched with the characteristic points of the prestored target control object, and identifying the control object in the interface image. Compared with a coordinate recording mode in the prior art, the method for identifying the target control object is more accurate and flexible, when the interface image is slightly changed, the position of the target control object is not influenced, and the operation control script cannot be failed to be executed due to the fact that the coordinate of the target control object in the interface is changed.
In the method for generating the operation script, the operation information of the target user is input into the operation intention identification model, and the operation intention information of the target user on each control object to be identified in the operation process is identified and obtained. The operation intention identification model is obtained by training a neural network model through operation information of a plurality of users and actual operation intention information of each user, the operation intention information of each to-be-identified control object is used for explaining the operation purpose of a target user to the to-be-identified control object, for each to-be-identified control object, the operation intention information of the to-be-identified control object is utilized to match and obtain a target control object corresponding to the to-be-identified control object from each control object in each interface image, then the target control object is associated with the operation intention information of the to-be-identified control object to obtain the operation intention information corresponding to the target control object, an operation control script corresponding to an operation flow is generated through the operation intention information corresponding to each target control object, and therefore the operation control script can identify and position each target control object in an equipment interface according to the time stamp sequence of the operation intention information corresponding to each target control object and execute corresponding operation intention on each target control object. Because the operation control script obtained in the embodiment of the application is executed according to the operation intention, compared with a control script which only repeats operation actions and movement tracks in the prior art, the operation control script in the embodiment of the application has higher reliability in achieving the operation purpose, and can accurately restore the flow of manual operation.
Based on the method for generating an operation script provided by the embodiment of the present application, the embodiment of the present application correspondingly discloses an apparatus for generating an operation script, which includes: an acquisition unit 401, an intention recognition unit 402, a matching unit 403, an association unit 404, and a generation unit 405.
The acquiring unit 401 is configured to acquire an interface image and a corresponding timestamp in an operation flow and acquire operation information of a target user in the operation flow in a process that the target user executes the operation flow.
Optionally, in a specific embodiment of the present application, the acquisition unit 401 includes:
and the triggering unit is used for triggering the screenshot of the interface when the target user executes the operation action each time, so that each interface image and the corresponding timestamp are obtained.
And an intention identifying unit 402, configured to input operation information of the target user into the operation intention identifying model, and identify and obtain operation intention information of the target user on each control object to be identified in the operation flow. The operation intention recognition model is obtained by training the neural network model through operation information of a plurality of users and actual operation intention information of each user, and the operation intention information of each control object to be recognized is used for explaining the operation purpose of the target user on the control object to be recognized.
And a matching unit 403, configured to match, for each control object to be identified, with the use of the operation intention information of the control object to be identified, to obtain a target control object corresponding to the control object to be identified from each control object in each interface image.
Optionally, in a specific embodiment of the present application, the operation intention information of the control object to be identified includes: the target user executes the operation intention action of the control to be recognized, the timestamp corresponding to the operation intention action and the coordinate corresponding to the operation intention action.
Wherein, the matching unit 403 includes: a first determining subunit and a second determining subunit.
And the first determining subunit is used for determining, for each control object to be identified, an interface image matched with the timestamp in the operation intention information from each interface image by using the timestamp in the operation intention information of the control object to be identified.
And the second determining subunit is used for determining a target control object matched with the coordinates in the operation intention information from each control object of the matched interface image by using the coordinates in the operation intention information of the control object to be identified.
The associating unit 404 is configured to associate the target control object with the operation intention information of the control object to be identified, so as to obtain operation intention information corresponding to the target control object.
The generating unit 405 is configured to generate an operation control script corresponding to the operation flow by using the operation intention information corresponding to each target control object. The operation control script is used for identifying each target control object positioned in the equipment interface by respectively utilizing the operation intention information corresponding to each target control object according to the time stamp sequence of the operation intention information corresponding to each target control object, and executing the corresponding operation intention on each target control object.
Optionally, in a specific embodiment of the present application, the method further includes: the device comprises a construction unit, an input unit and an adjusting unit.
And the construction unit is used for constructing a training data set. Wherein the training dataset comprises: operation information of a plurality of users and actual operation intention information of each user; the operation information comprises: the operation device comprises a plurality of operation actions, a time stamp corresponding to each operation action and coordinates corresponding to each operation action.
And the input unit is used for respectively inputting the operation information of each user in the training data set into the neural network model and respectively predicting to obtain the operation intention information of each user.
And the adjusting unit is used for continuously adjusting the weight and the threshold value in the neural network model according to the error between the operation intention information of each user obtained through prediction and the corresponding actual operation intention information until the error between the operation intention information of each user obtained by the adjusted neural network model and the corresponding actual operation information meets a preset convergence condition, and determining the adjusted neural network model as an operation intention identification model.
Optionally, in a specific embodiment of the present application, the method further includes:
and the identification unit is used for identifying the control object of each interface image and identifying each control object and the corresponding coordinate of the control object in the interface image.
Optionally, in a specific embodiment of the present application, the operating the control script includes: the operation intention information corresponding to each target control object and the identification algorithm adopted for identifying each target control object.
The specific principle and the execution process of the operation script generating device disclosed in the embodiment of the present application are the same as those of the operation script generating method disclosed in the embodiment of the present application, and reference may be made to corresponding parts in the operation script generating method disclosed in the embodiment of the present application, which are not described herein again.
In the device for generating the operation script provided by the embodiment of the application, the intention identifying unit 402 inputs the operation information of the target user into the operation intention identifying model, and identifies and obtains the operation intention information of the target user for each control object to be identified in the operation flow. The operation intention recognition model is obtained by training the neural network model through operation information of a plurality of users and actual operation intention information of each user, and the operation intention information of each control object to be recognized is used for explaining the operation purpose of the target user on the control object to be recognized. The matching unit 403 matches, for each control object to be identified, with the operation intention information of the control object to be identified, a target control object corresponding to the control object to be identified from each control object in each interface image, then the associating unit 404 associates the target control object with the operation intention information of the control object to be identified to obtain operation intention information corresponding to the target control object, and the generating unit 405 generates an operation control script corresponding to the operation flow by using the operation intention information corresponding to each target control object, so that the operation control script can identify, by using the operation intention information corresponding to each target control object according to the time stamp sequence of the operation intention information corresponding to each target control object, each target control object located in the device interface, and execute a corresponding operation intention on each target control object. Because the operation control script obtained in the embodiment of the application is executed according to the operation intention, compared with a control script which only repeats operation actions and movement tracks in the prior art, the operation control script in the embodiment of the application has higher reliability in achieving the operation purpose, and can accurately restore the flow of manual operation.
The present application discloses a computer-readable medium, on which a computer program is stored, wherein the program, when executed by a processor, implements a method of generating an operation script as described in the above embodiments.
The application also discloses an apparatus, comprising: one or more processors, a storage device, on which one or more programs are stored, the one or more programs, when executed by the one or more processors, causing the one or more processors to implement the method for generating an operation script as described in the above embodiments.
The previous description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the invention. Thus, the present invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
It is further noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrases "comprising a," "...," or "comprising" does not exclude the presence of additional like elements in a process, method, article, or apparatus that comprises the element.
Claims (10)
1. A method for generating an operation script is characterized by comprising the following steps:
acquiring an interface image and a corresponding timestamp in an operation flow and acquiring operation information of a target user in the operation flow in the process of executing the operation flow by the target user; the operation information of the target user refers to information for describing an operation performed by the target user in an operation flow; the operation information comprises: the method comprises the following steps of (1) carrying out a plurality of operation actions, a timestamp corresponding to each operation action and a coordinate corresponding to each operation action;
inputting the operation information of the target user into an operation intention recognition model, and recognizing to obtain the operation intention information of the target user on each control object to be recognized in the operation flow; the operation intention recognition model is obtained by training a neural network model through operation information of a plurality of users and actual operation intention information of each user; the operation intention information of each control object to be identified is used for explaining the operation purpose of the target user on the control object to be identified; the operation intention information does not concern the detail information of the repeated operation executed in the process of achieving the operation purpose; the operation intention information of the control object to be identified comprises the following steps: the target user performs an operation intention action on the control object to be recognized, executes a timestamp corresponding to the operation intention action, and executes a coordinate corresponding to the operation intention action;
for each control object to be identified, matching and obtaining a target control object corresponding to the control object to be identified from each control object in each interface image by using the operation intention information of the control object to be identified;
associating the target control object with the operation intention information of the control object to be identified to obtain operation intention information corresponding to the target control object;
generating an operation control script corresponding to the operation flow by using the operation intention information corresponding to each target control object; the operation control script is used for identifying each target control object positioned in an equipment interface by respectively utilizing the operation intention information corresponding to each target control object according to the time stamp sequence of the operation intention information corresponding to each target control object, and executing the corresponding operation intention on each target control object.
2. The method according to claim 1, wherein the obtaining, for each control object to be recognized, a target control object corresponding to the control object to be recognized by matching, from each control object in each interface image, the operation intention information of the control object to be recognized includes:
for each control object to be identified, determining an interface image matched with a timestamp in the operation intention information from each interface image by using the timestamp in the operation intention information of the control object to be identified;
and determining a target control object matched with the coordinates in the operation intention information from each control object of the matched interface image by using the coordinates in the operation intention information of the control object to be identified.
3. The method of claim 1, wherein the method for creating the operational intent recognition model comprises:
constructing a training data set; wherein the training data set comprises: operation information of a plurality of users and actual operation intention information of each of the users;
respectively inputting the operation information of each user in the training data set into a neural network model, and respectively predicting to obtain the operation intention information of each user;
continuously adjusting the weight and the threshold value in the neural network model according to the predicted error between the operation intention information of each user and the corresponding actual operation intention information until the error between the operation intention information of each user and the corresponding actual operation intention information obtained by the adjusted neural network model meets a preset convergence condition, and determining the adjusted neural network model as an operation intention identification model.
4. The method according to claim 2, before determining, from each control object of the matched interface image, a target control object that matches the coordinates in the operational intention information by using the coordinates in the operational intention information of the control object to be identified, further comprising:
and for each interface image, carrying out control object identification on the interface image, and identifying to obtain each control object and a control object corresponding coordinate in the interface image.
5. The method of claim 1, wherein the operation control script comprises: the operation intention information corresponding to each target control object and the identification algorithm adopted for identifying each target control object.
6. The method of claim 1, wherein the acquiring interface images and corresponding timestamps in the operational flow comprises:
and triggering the interface to capture the screen when the target user executes the operation action each time, so as to obtain each interface image and the corresponding timestamp.
7. An operation script generation apparatus, comprising:
the acquisition unit is used for acquiring an interface image and a corresponding timestamp in an operation flow and acquiring operation information of a target user in the operation flow in the process of executing the operation flow by the target user; the operation information of the target user refers to information for describing an operation performed by the target user in an operation flow; the operation information comprises: the method comprises the following steps of (1) carrying out a plurality of operation actions, a timestamp corresponding to each operation action and a coordinate corresponding to each operation action;
the intention identification unit is used for inputting the operation information of the target user into an operation intention identification model, and identifying and obtaining the operation intention information of the target user to each control object to be identified in the operation flow; the operation intention recognition model is obtained by training a neural network model through operation information of a plurality of users and actual operation intention information of each user; the operation intention information of each control object to be recognized is used for explaining the operation purpose of the target user on the control object to be recognized; the operation intention information does not concern the detail information of the repeated operation executed in the process of achieving the operation purpose; the operation intention information of the control object to be identified comprises the following steps: the target user performs an operation intention action on the control object to be recognized, executes a timestamp corresponding to the operation intention action, and executes a coordinate corresponding to the operation intention action;
the matching unit is used for matching each control object to be identified to obtain a target control object corresponding to the control object to be identified from each control object in each interface image by using the operation intention information of the control object to be identified;
the association unit is used for associating the target control object with the operation intention information of the control object to be identified to obtain the operation intention information corresponding to the target control object;
the generating unit is used for generating an operation control script corresponding to the operation flow by using the operation intention information corresponding to each target control object; the operation control script is used for identifying each target control object positioned in an equipment interface by respectively utilizing the operation intention information corresponding to each target control object according to the time stamp sequence of the operation intention information corresponding to each target control object, and executing the corresponding operation intention on each target control object.
8. The apparatus of claim 7, wherein the matching unit comprises:
the first determining subunit is configured to determine, for each control object to be identified, an interface image that matches the timestamp in the operation intention information from each interface image by using the timestamp in the operation intention information of the control object to be identified;
and the second determining subunit is used for determining a target control object matched with the coordinates in the operation intention information from each control object of the matched interface image by using the coordinates in the operation intention information of the control object to be identified.
9. A computer-readable medium, characterized in that a computer program is stored thereon, wherein the program, when being executed by a processor, is adapted to carry out the method of any one of claims 1 to 6.
10. An electronic device, comprising:
one or more processors;
a storage device having one or more programs stored thereon;
the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the method of any of claims 1-6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011580232.0A CN112578979B (en) | 2020-12-28 | 2020-12-28 | Method and device for generating operation script, readable medium and equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011580232.0A CN112578979B (en) | 2020-12-28 | 2020-12-28 | Method and device for generating operation script, readable medium and equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112578979A CN112578979A (en) | 2021-03-30 |
| CN112578979B true CN112578979B (en) | 2023-04-18 |
Family
ID=75140379
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011580232.0A Active CN112578979B (en) | 2020-12-28 | 2020-12-28 | Method and device for generating operation script, readable medium and equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112578979B (en) |
Family Cites Families (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9697012B2 (en) * | 2010-10-22 | 2017-07-04 | Hewlett Packard Enterprise Development Lp | Relation-based identification of automation objects |
| CN107193750B (en) * | 2017-07-04 | 2020-09-22 | 北京云测信息技术有限公司 | Script recording method and device |
| CN107608877B (en) * | 2017-08-11 | 2021-04-09 | 上海巍擎信息技术有限责任公司 | Automatic application program interface testing method and system based on machine learning |
| CN109783365A (en) * | 2018-12-15 | 2019-05-21 | 深圳壹账通智能科技有限公司 | Automated testing method, device, computer equipment and storage medium |
| CN109857674A (en) * | 2019-02-27 | 2019-06-07 | 上海优扬新媒信息技术有限公司 | A kind of recording and playback test method and relevant apparatus |
| CN110309059A (en) * | 2019-05-24 | 2019-10-08 | 深圳壹账通智能科技有限公司 | UI interface detection method, system, computer equipment and storage medium |
| CN111598128B (en) * | 2020-04-09 | 2023-05-12 | 腾讯科技（上海）有限公司 | Control state identification and control method, device, equipment and medium of user interface |
-
2020
- 2020-12-28 CN CN202011580232.0A patent/CN112578979B/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN112578979A (en) | 2021-03-30 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US8204310B2 (en) | Feature design for HMM based Eastern Asian character recognition | |
| CN110175236B (en) | Training sample generation method and device for text classification and computer equipment | |
| CN113255614A (en) | RPA flow automatic generation method and system based on video analysis | |
| CN109189519B (en) | Universal user desktop behavior simulation system and method | |
| JP6464504B6 (en) | Electronic device, processing method and program | |
| US8417026B2 (en) | Gesture recognition methods and systems | |
| CN104808794A (en) | Method and system for inputting lip language | |
| US11482001B2 (en) | Image processing device, image processing method, and image processing system | |
| US8149281B2 (en) | Electronic device and method for operating a presentation application file | |
| JP2014225135A (en) | Program, information processing device, and character recognition method | |
| CN102841676A (en) | Webpage browsing control system and method | |
| CN112578979B (en) | Method and device for generating operation script, readable medium and equipment | |
| JP6051991B2 (en) | Signal processing apparatus, signal processing method, and signal processing program | |
| KR20190027287A (en) | The method of mimesis for keyboard and mouse function using finger movement and mouth shape | |
| CN112163400A (en) | Information processing method and device | |
| JP2007220145A (en) | Online handwriting recognition apparatus | |
| CN114677767A (en) | Formula generation method and device, computing equipment and computer storage medium | |
| JP2018113643A (en) | Character string input device, input character string estimation method, and input character string estimation program | |
| CN114547231A (en) | Data tracing method and system | |
| Pan et al. | Research on functional test of mobile app based on robot | |
| WO2020124377A1 (en) | Modeling knowledge acquisition method and apparatus, and system | |
| CN117648298B (en) | Intelligent management system, method and device for tablet personal computer | |
| Roehm et al. | Towards identification of software improvements and specification updates by comparing monitored and specified end-user behavior | |
| JP2000163602A (en) | Input history storage device | |
| TWI571753B (en) | Electronic calculating apparatus for generating an interaction index map of an image, method thereof and computer program product thereof |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |