CN112346946B - User software operation behavior monitoring method and system based on control positioning - Google Patents
User software operation behavior monitoring method and system based on control positioning Download PDFInfo
- Publication number
- CN112346946B CN112346946B CN202011273420.9A CN202011273420A CN112346946B CN 112346946 B CN112346946 B CN 112346946B CN 202011273420 A CN202011273420 A CN 202011273420A CN 112346946 B CN112346946 B CN 112346946B
- Authority
- CN
- China
- Prior art keywords
- monitoring
- java
- user
- module
- mouse
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/34—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
- G06F11/3438—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment monitoring of user actions
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3003—Monitoring arrangements specially adapted to the computing system or computing system component being monitored
- G06F11/302—Monitoring arrangements specially adapted to the computing system or computing system component being monitored where the computing system component is a software system
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3051—Monitoring arrangements for monitoring the configuration of the computing system or of the computing system component, e.g. monitoring the presence of processing resources, peripherals, I/O links, software programs
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Computing Systems (AREA)
- Quality & Reliability (AREA)
- General Physics & Mathematics (AREA)
- Computer Hardware Design (AREA)
- Mathematical Physics (AREA)
- Debugging And Monitoring (AREA)
- Stored Programmes (AREA)
Abstract
The invention relates to a user software operation behavior monitoring method based on control positioning, which is mainly used for monitoring software operation behaviors of a user on a software program and recording operation information. The invention can obtain accurate and rich information by adopting a method based on control positioning. The method comprises two parts of a UIAutomation monitoring method and a Java monitoring method. The method solves the problems of inflexible monitoring information, high system environment requirement, incomplete applicable software variety and the like in the traditional method, and provides rich and accurate information for further operation.
Description
Technical Field
The invention belongs to the technical field of computer security, relates to user behavior monitoring, and particularly relates to a user software operation behavior monitoring method and system based on control positioning.
Background
At present, two schemes are mainly used for monitoring the operation behavior of user software, the first scheme is to monitor the absolute coordinate of the operation of a user in a computer, for example, the pixel position pressed by a mouse and the specific operation of the mouse are monitored, but although the scheme is simple and easy to implement, the recorded user behavior information has many disadvantages, for example, the recorded absolute coordinate is only specific to the current environment, when the system environment changes, for example, software is dragged, the specific operation of the user cannot be reversely pushed according to the absolute coordinate, so that the record becomes a useless record, and the robustness is lacked. The second scheme is a method for positioning a control, and the method can record a specific control target operated by a user, for example, which button the user clicks, and if the button object can be recorded, user software behavior monitoring under different environments can be realized. For example, microsoft provides an instect tool which can view control information of Native application programs, and a Java Swing framework provides a Java refer tool which views information of Java controls, but both the instect tool and the Java Swing framework are not open-source and cannot be developed secondarily on the basis of the instect tool, so that the effect of the scheme is limited.
Disclosure of Invention
In order to overcome the defects of the prior art, the invention aims to provide a method and a system for monitoring the operation behavior of user software based on control positioning, on one hand, the monitoring method is easy to deploy, has good compatibility, does not need to modify an application program, and is suitable for a Windows operating system; on the other hand, the monitoring method has good completeness and practicability, overcomes the problems of simple operation record, inflexible information recording and monitoring, high system environment requirement, incomplete applicable software types and the like in the traditional method, and provides rich and accurate information for further operation.
In order to achieve the purpose, the invention adopts the technical scheme that:
a user software operation behavior monitoring method based on control positioning comprises the following steps:
starting a monitoring program and appointing an application program to be monitored;
selecting UIAutomation monitoring or Java monitoring according to the type of the monitoring target, namely selecting Java monitoring if the path text input by the user contains jar, otherwise selecting UIAutomation monitoring.
UIAutomation monitoring uses Hook technology to monitor the operation of a user in a non-Java desktop application program;
java monitoring uses Java injection technology to monitor the operation of a user in a Java desktop application;
and storing the monitored user operation behavior information into a log, wherein the user software operation behavior refers to keyboard operation and mouse operation of a user, and the user operation behavior information refers to specific operation behavior of the user and information of an operated target control.
The Hook technology is that a callback function is set by using an interface provided by a system, a user triggers the callback function after executing keyboard operation and mouse operation, and then the callback function records user operation information; the Java injection technology is used for injecting monitoring codes into a Java virtual machine by using a Java Agent technology.
The invention also provides a user software operation behavior monitoring system based on control positioning, which comprises:
the UI module is used for setting system parameters and displaying monitored information;
the timing module is used for acquiring control information at the current mouse position of the user in a timing mode;
the process filtering module is used for monitoring process establishing behaviors and process quitting behaviors and maintaining a process linked list to be monitored;
the UIAutomation monitoring module records the operation behavior information of a user in a non-Java desktop application program;
the Java monitoring module is used for recording the operation behavior information of a user in a Java desktop application program;
and the log module is used for storing the operation behavior information of the user into a log.
In the invention, the UI module, the timing module, the process filtering module, the UIAutomation monitoring module and the log module belong to a main program and are compiled by using C + + and Qt, and the Java monitoring module is an independent Jar package and is compiled by using Java.
Furthermore, the UI module is divided into a process control sub-module, an information display sub-module and a control identification sub-module, wherein the process control sub-module is used for controlling a monitoring process and comprises a Record button, a Pause button, a Finish button, a Save button, a mode selection control and a file path selection control; the information display submodule is used for displaying information of a control at the mouse suspension position and monitored operation information and comprises a user operation information list and a control information list, and the control identification submodule is used for identifying a control area at the mouse suspension position by a red frame; the interface of the UI module supports the user to specify the application program to be monitored in two ways:
(1) a user designates an application program to be monitored in a process list, starts the corresponding application program and monitors the dynamic loading behavior of the designated application program to be monitored and the subprocess thereof;
(2) the user directly inputs the path and relevant parameters of the application program to be monitored, starts the corresponding application program and monitors the dynamic loading behavior of the application program and the subprocess of the application program to be monitored.
Further, the timing module is implemented as follows: firstly, registering a window on Windows, and then defining a window message processing function; setting a timing function by adopting QTimer aiming at a Native mode, setting a timing function by adopting a Timer class aiming at a Java mode, and refreshing a window at fixed time; when the system starts to run, timing functions corresponding to different modes start to execute, a Rectangle object of the control is obtained according to the position of mouse operation, and finally a red frame window is refreshed according to Rectangle object information in a SetWindowPos mode.
Furthermore, the process filtering module is divided into a process creating behavior monitoring submodule, a process quitting behavior monitoring submodule and a process recording submodule, the process creating behavior and the process quitting behavior in the monitoring system are monitored, a monitoring process list is maintained, operations except for a monitoring target are filtered out according to the Pid, the process creating behavior monitoring submodule and the process quitting behavior monitoring submodule use WMI technology, firstly, a COM interface of WMI is initialized, then, a process creating behavior monitoring callback function and a process quitting behavior callback function are set according to wql statements provided by WMI, and a process monitoring list is maintained in the callback function.
Further, the UIAutomation monitoring module is divided into three parts, namely an UIAutomation mouse monitoring sub-module, an UIAutomation keyboard monitoring sub-module and an UIAutomation data processing sub-module, wherein:
the UIAutomation mouse monitoring sub-module is divided into two parts in total, information is sent to the main program in the mouse monitoring function, then the information is returned directly without transmitting the mouse information downwards, and a virtual mouse information is generated at the same position after the processing in the main program is finished; after receiving the sent message, the main program judges whether the operation is mouse operation on the appointed application program through a process filtering module, if not, the main program does not process the operation, if so, a mouse position control object is obtained, mouse action is simulated to generate a virtual mouse message, and the mouse message is transmitted downwards;
the execution flow of the UIAutomation keyboard monitoring submodule is divided into two parts, a keyboard operation callback function is set by using a system API SetWindowEventHook provided by Windows, and the callback function in a Dll file is triggered when a user performs keyboard operation after monitoring is started; after receiving the message, the main program acquires the control currently obtaining the focus by using a GetFocusedElement function, judges whether the two controls are the same by using CompareElements, if so, updates the text value of the control in the current state of the user, otherwise, adds an action to determine the operation type, and finally delivers the operation type to the data processing sub-module for processing;
and the UIAutomation data processing submodule synthesizes the monitoring records of the mouse monitoring submodule into complete operation and displays the complete operation in a user operation information list.
Furthermore, the invention also comprises an injection module for injecting the monitoring program code into the Java virtual machine, wherein the code is a program for monitoring the operation of the user on the Java application program, the Java application program runs on the Java virtual machine, the Java execution injection module is started by using QPprocess, then the attach method in VirtualMachine is used for attaching to the target process, finally the target process is loaded with the Jar package by using the loadAgent method of VirtualMachine, and finally the monitoring program is injected into the virtual machine to monitor the operation of the user on the Java application program.
Furthermore, the Java monitoring module comprises two parts, wherein the first part is a Java data receiving submodule and a Java data processing submodule and is positioned in the main program; the second part is a Java keyboard monitoring submodule, a Java mouse monitoring submodule and a Java data sending submodule which are positioned in the injected Jar package; wherein:
the Java mouse monitoring submodule is used for monitoring mouse operation in a Java program, acquiring control information and mouse operation information corresponding to the operation after the operation is monitored, and the mouse operation information and the control information are realized by using a mouse EventListener function provided by Java;
the Java keyboard monitoring submodule is used for monitoring keyboard operation in a Java program, acquiring control information and keyboard operation information corresponding to the operation after the operation is monitored, and providing addAWTventLister by using Java to set FocusEventLister; judging whether the control is a text box when the focus of the control is lost, and judging whether the text of the control is changed with the text obtained the focus last time if the control is the text box;
the Java data sending submodule is used for formatting logs and sending data to the main program and is realized by using a UDP packet provided by Java;
the Java data receiving submodule is used for receiving UDP data sent by Java monitoring, the UDP data is realized by using QUdpSocket provided by Qt, a ReadPendingDatagrams function is bound to a readyRead signal, the data is received, when a UDP end receives the data, firstly, a str type json is processed into a map, secondly, whether the data is the data sent by the polling submodule is judged, if yes, the position and size information of a control in the map is obtained, the information is displayed through the control identification submodule and the information display submodule, and if not, the data enters the data processing submodule to process the data;
the Java data processing submodule is used for synthesizing the monitoring records of the Java mouse monitoring submodule and the Java keyboard monitoring submodule into a complete operation.
Compared with the prior art, the invention has the beneficial effects that:
1. the WMI technology is adopted to realize the monitoring of the process and the sub-processes thereof, and all additionally started processes of the application program in the running process can be monitored simultaneously.
2. C + + and Java mixed programming is used, monitoring of the Java virtual machine is achieved through the framework, the monitoring mode is automatically selected according to the monitoring target, and the problem that Native application programs and Java cannot be uniformly managed and monitored due to the virtual machine is solved.
3. The UDP communication scheme is used for solving the problem that the virtual machine and the main program cannot communicate when the operation levels are different.
4. The virtual key technology is used for solving the problem of information acquisition failure caused by the fact that mouse response precedes information acquisition.
5. The real-time positioning technology is used, the position of the mouse of the user and control information are captured in real time and displayed in the main program, and user experience is improved.
Drawings
FIG. 1 is an overall step diagram of a user software operation behavior monitoring method based on control positioning according to the present invention.
FIG. 2 is an overall architecture diagram of the user software operation behavior monitoring system based on control positioning according to the present invention.
FIG. 3 is a flow chart of the timing module of the present invention.
FIG. 4 is a flow diagram of the process filter module of the present invention.
Fig. 5 is a flowchart of the UIAutomation mouse monitoring sub-module (a process of sending a message to the main program) of the present invention.
FIG. 6 is a flowchart of the UIAutomation mouse monitoring sub-module (the processing flow after the main program receives the message) of the present invention.
Fig. 7 is a flowchart of the UIAutomation keyboard monitoring sub-module of the present invention (a flow of sending a message to the main program).
Fig. 8 is a flowchart of the UIAutomation keyboard monitoring sub-module of the present invention (processing flow after the main program receives the message).
FIG. 9 is a flow chart of the UIAutomation data processing sub-module of the present invention.
FIG. 10 is an overall architecture diagram of the Java monitor module according to the present invention.
FIG. 11 is a flowchart of the Java mouse monitor submodule according to the present invention.
FIG. 12 is a flowchart of the Java keyboard monitor submodule according to the present invention.
FIG. 13 is a flowchart of the Java data transmitting submodule according to the present invention.
FIG. 14 is a flow chart of the Java data receiving submodule according to the present invention.
Detailed Description
The embodiments of the present invention will be described in detail below with reference to the drawings and examples.
The invention relates to a method for monitoring user software operation behaviors based on control positioning, wherein the user software operation behaviors refer to keyboard operation and mouse operation of a user, and the key point is to monitor the software operation behaviors of the user on a software program and record operation information at the same time. The invention can obtain accurate and rich information by adopting a method based on control positioning. The method comprises two parts of UIAutomation monitoring and Java monitoring. As shown in fig. 1, the monitoring method of the present invention includes the following steps:
a monitoring program (UIAutomation monitoring or Java monitoring) is started, specifying the application to be monitored.
Selecting UIAutomation monitoring or Java monitoring according to the type of the monitoring target, namely selecting Java monitoring if the path text input by the user contains jar, otherwise selecting UIAutomation monitoring.
UIAutomation monitors the operation of a user in a non-Java desktop application program by using a Hook technology, the Hook technology is that a callback function is set by using an interface provided by a system, the user triggers the callback function after executing keyboard operation and mouse operation, and then the callback function records user operation information, wherein the callback function is a function which is called by another party when a specific event or condition occurs and is used for responding to the event or condition.
Java monitoring uses Java injection technology to monitor user operations in Java desktop applications, which is the injection of monitoring code into a Java virtual machine using Java Agent technology.
And storing the monitored user operation behavior information into a log, wherein the user operation behavior information refers to the specific operation behavior of the user and the information of the target control of the operation.
Referring to fig. 2, the present invention also provides a corresponding monitoring system, which mainly includes: the system comprises a UI module, a timing module, a process filtering module, a monitoring module (UIAutomation monitoring module and Java monitoring module), a log module, an injection module and the like. The UI module, the timing module, the process filtering module, the UIAutomation monitoring module and the log module belong to a main program, and are written by using C + + and Qt, and the injection module and the Java monitoring module are independent Jar packages and are written by using Java.
The UI module is used for setting system parameters and displaying monitored information, is an interface of a main program and is divided into a process control submodule, an information display submodule and a control identification submodule, wherein the process control submodule is used for controlling a monitoring process and comprises a Record button, a Pause button, a Finish button, a Save button, a mode selection, a file path selection and other configuration controls. The information display sub-module is used for displaying information of the control at the mouse suspension position and monitored operation information, and comprises a user operation information list and a control information list. And the control identification submodule is used for identifying a control area at the mouse suspension position by using a red frame.
The UI module is supported on the interface, and the user specifies the application program to be monitored in two ways:
(1) a user designates an application program to be monitored in a process list, starts the corresponding application program and monitors the dynamic loading behavior of the designated application program to be monitored and the subprocess thereof;
(2) the user directly inputs the path and relevant parameters of the application program to be monitored, starts the corresponding application program and monitors the dynamic loading behavior of the application program and the subprocess of the application program to be monitored.
The timing module is used for acquiring control information at the current mouse position of the user at regular time, feeding the control information back to the main program and displaying the control information in the information display interface. Referring to fig. 3, the timing module is implemented as follows: firstly, registering a window on Windows, and then defining a window message processing function; setting a timing function by adopting QTimer aiming at a Native mode, setting a timing function by adopting a Timer class aiming at a Java mode, and refreshing a window at fixed time; when the system starts to run, timing functions corresponding to different modes start to execute, a Rectangle object of the control is obtained according to the position of mouse operation, and finally a red frame window is refreshed according to Rectangle object information in a SetWindowPos mode.
The process filtering module is used for monitoring a process creation behavior and a process exit behavior and maintaining a process linked list to be monitored, and referring to fig. 4, the process filtering module is divided into a process creation behavior monitoring submodule, a process exit behavior monitoring submodule and a process recording submodule. The functions are to create behaviors and quit behaviors for the processes in the monitoring system, maintain a monitoring process list and filter out operations except for monitoring targets according to the Pid. The process creating behavior monitoring submodule and the process quitting behavior monitoring submodule use WMI technology in the implementation process, firstly, a COM interface of WMI is initialized, then, a process creating behavior monitoring callback function and a process quitting behavior callback function are set according to wql statements provided by WMI, and a process monitoring list is maintained in the callback function.
The UIAutomation monitoring module is used for recording operation behavior information of a user in a non-Java desktop application program, and is divided into an UIAutomation mouse monitoring submodule, an UIAutomation keyboard monitoring submodule and an UIAutomation data processing submodule, wherein:
the execution flow of the UIAutomation mouse monitoring submodule is divided into two parts, according to the system requirements, a mouse operation monitoring callback function needs to be placed in a Dll file (hookdll), and the callback function in the Dll file is triggered when a user performs mouse operation after monitoring is started. The UIAutomation interface can not be used in the mouse callback function and the keyboard callback function after testing, so that a message needs to be sent to the main program, and the UIAutomation interface is used in the main program. Therefore, it is to be ensured that the control information is obtained by using the UIAutomation interface and then the mouse message is transmitted, otherwise, the target program receives the mouse message first and makes a response, and then the information obtained by using the UIAutomation interface is inaccurate, for example, a window is closed.
In summary, the basic flow of mouse processing is that a message is sent to a main program in a mouse monitoring function, then the message is returned directly without transferring the mouse message downwards, and a virtual mouse message is generated at the same position after the processing in the main program is completed. The flow is shown in fig. 5.
After receiving the sent message, the main program judges whether the operation is the mouse operation on the appointed application program through the process filtering module, if not, the processing is not carried out, if so, the mouse position control object is obtained, the mouse action is simulated to generate a virtual mouse message, and the mouse message is transmitted downwards. The process flow is shown in fig. 6.
Referring to fig. 7 and 8, as with the UIAutomation mouse monitoring sub-module, the execution flow of the UIAutomation keyboard monitoring sub-module is divided into two parts, a system API SetWindowEventHook provided by Windows is used to set a keyboard operation callback function, after monitoring is started, a user triggers the callback function in a Dll file when performing keyboard operation, and the processing flow is as shown in fig. 7. After receiving the message, the main program obtains the control currently obtaining the focus by using a getfocusedlelement function, judges whether the two controls are the same by using CompareElements, if so, updates the text value of the control in the current state of the user, otherwise, adds an action to determine the operation type, and finally delivers the operation type to the data processing sub-module for processing, wherein the processing flow is as shown in fig. 8.
Referring to fig. 9, the UIAutomation data processing sub-module synthesizes the monitoring records of the mouse monitoring sub-module into a complete operation and displays the complete operation in the user operation information list. For example, a mouse left button press and a mouse left button lift with the same coordinates are combined into a click operation. After entering the module, mouse operation is synthesized according to the state machine, then the synthesized operation input information display sub-module is displayed, and meanwhile, the state machine can also be used for processing unexpected situations, such as pressing of a left key and then pressing of a right key.
The injection module is used for injecting the monitoring program code into the Java virtual machine, and since the main program is a C + + program and cannot directly acquire control information of the Java application, the monitoring module needs to be injected into the target Java application by using a Java injection technology. The injection implementation method comprises the steps of firstly starting a Java execution injection module by using QProcess, then attaching an attach method in a VirtualMachine to a target process, finally loading a Jar package by the target process by using a loadAgent method of the VirtualMachine, and finally injecting a monitoring program to the virtual machine to monitor the operation of a user on the Java application program.
The Java monitoring module is used for recording operation behavior information of a user in a Java desktop application program and recording the Java program, and referring to FIG. 10, the Java monitoring module comprises two parts, wherein the first part is a Java data receiving sub-module and a Java data processing sub-module, is positioned in a main program and is compiled by C + +; the second part is a Java keyboard monitoring submodule, a Java mouse monitoring submodule and a Java data sending submodule which are positioned in the injected Jar package.
Referring to fig. 11, the Java mouse monitoring submodule is configured to monitor a mouse operation in a Java program, acquire control information and mouse operation information corresponding to the operation after the operation is monitored, and use a MouseEventListener function provided by Java to implement the mouse operation.
Referring to fig. 12, the Java keyboard monitoring sub-module is used to monitor the keyboard operation in the Java program and obtain the control information and the keyboard operation information corresponding to the operation after the operation is monitored, but due to the particularity of Java, this sub-module is not implemented using the keyboard monitoring function provided by Java. Through tests, the keyboard behavior cannot be monitored after the Chinese input method is started by using keyavent and keyboardfocusmanager, and the KeyEvent cannot be monitored because the keyboard behavior can only be responded by pressing two keys simultaneously. Using Java to provide addaweventlistener to set FocusEventListener; and judging whether the control is a text box when the focus of the control is lost, and judging whether the text of the control is changed with the text obtained the focus last time if the control is the text box.
Referring to fig. 13, the Java data sending sub-module is configured to format the log and send the data to the host program, and may be implemented using a UDP packet provided by Java.
Referring to fig. 14, the Java data receiving submodule is configured to receive UDP data sent from Java monitoring, and may specifically be implemented using QUdpSocket provided by Qt, and binds a readpending datagrams function to a readyRead signal, and receives the data, when the UDP end receives the data, first processes json of a str type into a map, and then determines whether the data is the data sent by the polling submodule, if so, obtains position and size information of a control in the map, displays the information through the control identification submodule and the information display submodule, and otherwise enters the data processing submodule to process the data.
The Java data processing submodule is used for synthesizing monitoring records of the Java mouse monitoring submodule and the Java keyboard monitoring submodule into a complete operation, and the processing process is the same as that of the UIAutomation data processing submodule.
The log module is used for storing the operation behavior information of the user into a log, the user operation can be stored into a script file by adopting a file storage module, and the file storage module consists of two submodules, namely a UIautomation application monitoring file storage submodule and a Java monitoring file storage submodule.
Claims (8)
1. A user software operation behavior monitoring method based on control positioning is characterized by comprising the following steps:
starting a monitoring program and appointing an application program to be monitored;
selecting UIAutomation monitoring or Java monitoring according to the type of the monitoring target, namely selecting Java monitoring if the path text input by the user contains Java monitoring, otherwise selecting UIAutomation monitoring;
UIAutomation monitoring uses Hook technology to monitor the operation of a user in a non-Java desktop application program;
java monitoring uses Java injection technology to monitor the operation of a user in a Java desktop application;
storing the monitored user operation behavior information into a log, wherein the user software operation behavior refers to keyboard operation and mouse operation of a user, and the user operation behavior information refers to specific operation behavior of the user and information of an operated target control;
wherein, the user specifies the application to be monitored in two ways:
(1) a user designates an application program to be monitored in a process list, starts the corresponding application program and monitors the dynamic loading behavior of the designated application program to be monitored and the subprocess thereof;
(2) a user directly inputs a path and related parameters of an application program to be monitored, starts the corresponding application program and monitors the dynamic loading behavior of the application program and the subprocess of the application program to be monitored;
the control information at the current mouse position of the user is regularly acquired by the following method:
firstly, registering a window on Windows, and then defining a window message processing function; setting a timing function by adopting QTimer aiming at a Native mode, setting a timing function by adopting Timer class aiming at a Java mode, and refreshing a window at regular time; when the system starts to operate, timing functions corresponding to different modes start to be executed, a Rectangle object of the control is obtained according to the position of mouse operation, and finally the red frame window is refreshed according to Rectangle object information in a SetWindowPos mode.
2. The method for monitoring the operation behavior of the user software based on the control positioning as claimed in claim 1, wherein the Hook technology is to use an interface provided by a system to set a callback function, a user triggers the callback function after performing keyboard operation and mouse operation, and then the callback function records the user operation information; the Java injection technology is used for injecting monitoring codes into a Java virtual machine by using a Java Agent technology.
3. A user software operation behavior monitoring system based on control positioning is characterized by comprising:
the UI module is used for setting system parameters and displaying monitored information;
the timing module is used for acquiring control information at the current mouse position of the user in a timing mode;
the process filtering module is used for monitoring process establishing behaviors and process quitting behaviors and maintaining a process linked list to be monitored;
the UIAutomation monitoring module records the operation behavior information of a user in a non-Java desktop application program;
the Java monitoring module is used for recording the operation behavior information of a user in a Java desktop application program;
the log module is used for storing the operation behavior information of the user into a log;
the UI module is divided into a process control sub-module, an information display sub-module and a control identification sub-module, wherein the process control sub-module is used for controlling a monitoring process and comprises a Record button, a Pause button, a Finish button, a Save button, a mode selection control and a file path selection control; the information display submodule is used for displaying information of a control at the mouse suspension position and monitored operation information and comprises a user operation information list and a control information list, and the control identification submodule is used for identifying a control area at the mouse suspension position by a red frame; the interface of the UI module supports the user to specify the application program to be monitored in two ways:
(1) a user designates an application program to be monitored in a process list, starts the corresponding application program and monitors the dynamic loading behavior of the designated application program to be monitored and the subprocess thereof;
(2) directly inputting the path and related parameters of the application program to be monitored by a user, starting the corresponding application program, and monitoring the dynamic loading behavior of the specified application program to be monitored and the subprocess thereof;
the timing module is implemented as follows: firstly, registering a window on Windows, and then defining a window message processing function; setting a timing function by adopting QTimer aiming at a Native mode, setting a timing function by adopting a Timer class aiming at a Java mode, and refreshing a window at fixed time; when the system starts to run, timing functions corresponding to different modes start to execute, a Rectangle object of the control is obtained according to the position of mouse operation, and finally a red frame window is refreshed according to Rectangle object information in a SetWindowPos mode.
4. The system for monitoring the operation behavior of the user software based on the control positioning as claimed in claim 3, wherein the UI module, the timing module, the process filtering module, the UIAutomation monitoring module and the log module belong to a main program and are written by using C + + and Qt, and the Java monitoring module is an independent Jar package and is written by using Java.
5. The system for monitoring the operation behavior of the user software based on the control positioning as claimed in claim 3, wherein the process filtering module is divided into a process creation behavior monitoring submodule, a process exit behavior monitoring submodule and a process recording submodule, the process creation behavior and the process exit behavior in the monitoring system are monitored, a monitoring process list is maintained, and operations except for a monitoring target are filtered out according to the Pid, wherein the process creation behavior monitoring submodule and the process exit behavior monitoring submodule use WMI technology to initialize a COM interface of WMI, then set a process creation behavior monitoring callback function and a process exit behavior callback function according to wql statements provided by WMI, and maintain a process monitoring list in the callback function.
6. The system for monitoring the operation behavior of the user software based on the control positioning as claimed in claim 3, wherein the UIAutomation monitoring module is divided into three parts, namely an UIAutomation mouse monitoring sub-module, an UIAutomation keyboard monitoring sub-module and an UIAutomation data processing sub-module, wherein:
the execution flow of the UIAutomation mouse monitoring submodule is divided into two parts, a message is sent to a main program in a mouse monitoring function, then the message is directly returned without transmitting the mouse message downwards, and a virtual mouse message is generated at the same position after the processing in the main program is finished; after receiving the sent message, the main program judges whether the operation is mouse operation on the appointed application program through a process filtering module, if not, the main program does not process the operation, if so, a mouse position control object is obtained, mouse action is simulated to generate a virtual mouse message, and the mouse message is transmitted downwards;
the execution flow of the UIAutomation keyboard monitoring submodule is divided into two parts, a keyboard operation callback function is set by using a system API SetWindowEventHook provided by Windows, and the callback function in a Dll file is triggered when a user performs keyboard operation after monitoring is started; after receiving the message, the main program acquires the control currently obtaining the focus by using a GetFocusedElement function, judges whether the two controls are the same by using CompareElements, if so, updates the text value of the control in the current state of the user, otherwise, adds an action to determine the operation type, and finally delivers the operation type to the data processing sub-module for processing;
and the UIAutomation data processing submodule synthesizes the monitoring records of the mouse monitoring submodule into complete operation and displays the complete operation in a user operation information list.
7. The system for monitoring the operation behavior of the user software based on the control positioning as recited in claim 3, further comprising an injection module for injecting a monitoring program code into the Java virtual machine, wherein the code is a program for monitoring the operation of the user on a Java application program, the Java application program runs on the Java virtual machine, the Java execution injection module is started by using qpprocesses, then an attach method in VirtualMachine is used for attaching to the target process, finally a loadAgent method in VirtualMachine is used for loading the target process into the Jar package, and finally the monitoring program is injected into the virtual machine to monitor the operation of the user on the Java application program.
8. The system for monitoring the operation behavior of the user software based on the control positioning as claimed in claim 3, wherein the Java monitoring module comprises two parts, the first part is a Java data receiving sub-module and a Java data processing sub-module, and is located in the main program; the second part is a Java keyboard monitoring submodule, a Java mouse monitoring submodule and a Java data sending submodule which are positioned in the injected Jar packet; wherein:
the Java mouse monitoring submodule is used for monitoring mouse operation in a Java program, acquiring control information and mouse operation information corresponding to the operation after the operation is monitored, and the mouse operation information and the control information are realized by using a mouse EventListener function provided by Java;
the Java keyboard monitoring submodule is used for monitoring keyboard operation in a Java program, acquiring control information and keyboard operation information corresponding to the operation after the operation is monitored, and providing addAWTventLister by using Java to set FocusEventLister; judging whether the control is a text box when the focus of the control is lost, and judging whether the text of the control is changed with the text obtained the focus last time if the control is the text box;
the Java data sending submodule is used for formatting logs and sending data to the main program and is realized by using a UDP packet provided by Java;
the Java data receiving submodule is used for receiving UDP data sent by Java monitoring, the UDP data receiving submodule is realized by using QUdpSocket provided by Qt, a ReadPendingDatagrams function is bound to a readyRead signal, the data is received, when a UDP end receives the data, a str type json is firstly processed into a map, whether the data is the data sent by the polling submodule is judged, if yes, the position and size information of a control in the map is obtained, the information is displayed through the control identification submodule and the information display submodule, and if not, the data enters the data processing submodule to process the data;
the Java data processing submodule is used for synthesizing the monitoring records of the Java mouse monitoring submodule and the Java keyboard monitoring submodule into a complete operation.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011273420.9A CN112346946B (en) | 2020-11-13 | 2020-11-13 | User software operation behavior monitoring method and system based on control positioning |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011273420.9A CN112346946B (en) | 2020-11-13 | 2020-11-13 | User software operation behavior monitoring method and system based on control positioning |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112346946A CN112346946A (en) | 2021-02-09 |
| CN112346946B true CN112346946B (en) | 2022-06-21 |
Family
ID=74363832
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011273420.9A Active CN112346946B (en) | 2020-11-13 | 2020-11-13 | User software operation behavior monitoring method and system based on control positioning |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112346946B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113395360A (en) * | 2021-08-17 | 2021-09-14 | 苏州浪潮智能科技有限公司 | Windows system user operation simulation method, system, terminal and storage medium |
| CN113779570B (en) * | 2021-09-18 | 2024-02-23 | 深信服科技股份有限公司 | Security protection method and device based on core process and electronic equipment |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101996106A (en) * | 2010-12-17 | 2011-03-30 | 南京中兴力维软件有限公司 | Method for monitoring software running state |
| CN106649024A (en) * | 2016-09-22 | 2017-05-10 | 微梦创科网络科技（中国）有限公司 | Method and device for real-time monitoring of application performance |
| CN107273145A (en) * | 2016-04-06 | 2017-10-20 | 中兴通讯股份有限公司 | A kind of method of managing software and device, terminal |
| CN111831395A (en) * | 2020-07-09 | 2020-10-27 | 西安交通大学 | Behavior monitoring and analyzing method and system |
Family Cites Families (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108074626A (en) * | 2016-11-09 | 2018-05-25 | 长沙博为软件技术股份有限公司 | A kind of Windows form controls data grab method |
| US10223138B2 (en) * | 2017-03-31 | 2019-03-05 | International Business Machines Corporation | Software installation assistance method and system |
| CN107885433B (en) * | 2017-11-23 | 2021-07-27 | Oppo广东移动通信有限公司 | Control method and device for terminal equipment, terminal, server and storage medium |
| CN109344616B (en) * | 2018-09-13 | 2020-12-08 | 西安交通大学 | Method and device for monitoring dynamic loading behavior of mobile application program |
-
2020
- 2020-11-13 CN CN202011273420.9A patent/CN112346946B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101996106A (en) * | 2010-12-17 | 2011-03-30 | 南京中兴力维软件有限公司 | Method for monitoring software running state |
| CN107273145A (en) * | 2016-04-06 | 2017-10-20 | 中兴通讯股份有限公司 | A kind of method of managing software and device, terminal |
| CN106649024A (en) * | 2016-09-22 | 2017-05-10 | 微梦创科网络科技（中国）有限公司 | Method and device for real-time monitoring of application performance |
| CN111831395A (en) * | 2020-07-09 | 2020-10-27 | 西安交通大学 | Behavior monitoring and analyzing method and system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112346946A (en) | 2021-02-09 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108763086B (en) | Script recording method, terminal and computer readable storage medium based on remote true machine | |
| CN102419732B (en) | Android-platform-based automated testing method and testing tool | |
| CN107797904B (en) | Method and terminal for measuring response time | |
| CN112346946B (en) | User software operation behavior monitoring method and system based on control positioning | |
| CN110362483B (en) | Performance data acquisition method, device, equipment and storage medium | |
| JP2018535459A (en) | Robotic process automation | |
| CN108509343A (en) | Automated testing method based on image recognition technology and system | |
| US20170322637A1 (en) | Mouse action capture and replay system and method, recording medium, and program product | |
| CN111026645A (en) | User interface automatic testing method and device, storage medium and electronic equipment | |
| Wang et al. | Automatic Android GUI traversal with high coverage | |
| US20140361991A1 (en) | Method and electronic device for controlling mouse module | |
| EP2731015B1 (en) | Computer infrastructure performance system | |
| US10303349B2 (en) | Image-based application automation | |
| CN107704376B (en) | Game automation testing method and system | |
| CN107102937B (en) | User interface testing method and device | |
| CN109344058B (en) | Automatic small program testing method | |
| CN111694744B (en) | Test development method based on monkey source code | |
| US11334240B2 (en) | Method, device, electronic device, and storage medium for sending and receiving message | |
| CN111382312B (en) | Post-class problem display method, device, equipment and medium based on WeChat public number | |
| CN114979730A (en) | Screen transmission interaction method and device, computer equipment and storage medium | |
| CN116400846A (en) | Realization method, equipment and medium for simulating touch operation 3D application | |
| CN113093981B (en) | Method and equipment for generating long graphs of operation steps by screen recording of Android terminal | |
| CN107391354B (en) | Method and system for automatically monitoring terminal performance and storage device | |
| JP5266416B1 (en) | Test system and test program | |
| CN114911402A (en) | Dragging interaction method and system between remote application and local system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |