CN116991659A - Virtual training system-oriented data collection system and method - Google Patents
Virtual training system-oriented data collection system and method Download PDFInfo
- Publication number
- CN116991659A CN116991659A CN202310752938.8A CN202310752938A CN116991659A CN 116991659 A CN116991659 A CN 116991659A CN 202310752938 A CN202310752938 A CN 202310752938A CN 116991659 A CN116991659 A CN 116991659A
- Authority
- CN
- China
- Prior art keywords
- module
- data
- virtual training
- sub
- data collection
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000013480 data collection Methods 0.000 title claims abstract description 108
- 238000000034 method Methods 0.000 title claims abstract description 106
- 238000007781 pre-processing Methods 0.000 claims abstract description 59
- 230000008569 process Effects 0.000 claims abstract description 58
- 238000012549 training Methods 0.000 claims abstract description 40
- 238000004891 communication Methods 0.000 claims abstract description 9
- 238000012545 processing Methods 0.000 claims abstract description 9
- 238000007726 management method Methods 0.000 claims description 27
- 230000006399 behavior Effects 0.000 claims description 26
- 238000012544 monitoring process Methods 0.000 claims description 25
- 238000004140 cleaning Methods 0.000 claims description 18
- 238000004590 computer program Methods 0.000 claims description 12
- 238000013500 data storage Methods 0.000 claims description 11
- 238000004088 simulation Methods 0.000 claims description 9
- 238000010276 construction Methods 0.000 claims description 6
- 238000012360 testing method Methods 0.000 claims description 5
- 238000013481 data capture Methods 0.000 claims description 3
- 238000010586 diagram Methods 0.000 description 10
- 230000006870 function Effects 0.000 description 4
- 238000000605 extraction Methods 0.000 description 3
- 238000004458 analytical method Methods 0.000 description 2
- 238000004422 calculation algorithm Methods 0.000 description 2
- 230000010354 integration Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 238000005457 optimization Methods 0.000 description 2
- 238000006243 chemical reaction Methods 0.000 description 1
- 238000013136 deep learning model Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- 238000011900 installation process Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000003062 neural network model Methods 0.000 description 1
- 238000011084 recovery Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000001960 triggered effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3089—Monitoring arrangements determined by the means or processing involved in sensing the monitored data, e.g. interfaces, connectors, sensors, probes, agents
- G06F11/3093—Configuration details thereof, e.g. installation, enabling, spatial arrangement of the probes
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3003—Monitoring arrangements specially adapted to the computing system or computing system component being monitored
- G06F11/3006—Monitoring arrangements specially adapted to the computing system or computing system component being monitored where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3003—Monitoring arrangements specially adapted to the computing system or computing system component being monitored
- G06F11/302—Monitoring arrangements specially adapted to the computing system or computing system component being monitored where the computing system component is a software system
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3065—Monitoring arrangements determined by the means or processing involved in reporting the monitored data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/34—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
- G06F11/3438—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment monitoring of user actions
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2201/00—Indexing scheme relating to error detection, to error correction, and to monitoring
- G06F2201/80—Database-specific techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2201/00—Indexing scheme relating to error detection, to error correction, and to monitoring
- G06F2201/865—Monitoring of software
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Quality & Reliability (AREA)
- General Physics & Mathematics (AREA)
- Computing Systems (AREA)
- Mathematical Physics (AREA)
- Computer Hardware Design (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The invention provides a data collection system and a data collection method for a virtual training system, which belong to the technical field of data processing, wherein the system comprises: the system comprises a collection module and a preprocessing module which are in communication connection; the collection module is used for carrying out page traversal and behavior tracking on the virtual training process of each operation terminal to obtain information data and data screenshot of each operation terminal; and the preprocessing module is used for acquiring the preprocessing data of each operation terminal according to the information data and the data screenshot of each operation terminal. According to the data collection system and method for the virtual training system, the problem of data island in training application can be effectively solved through automatic non-invasive data collection and arrangement into the preprocessing data collection interface convenient to call, the data collection system and method for the virtual training system are suitable for virtual training application and web side, and can support data collection of various types and be widely applied to various application scenes.
Description
Technical Field
The invention relates to the technical field of data processing, in particular to a data collection system and method for a virtual training system.
Background
How to convert training data into information and knowledge for guiding and optimizing the training process is a main bottleneck for improving the training effect. The development of the data warehouse technology (ETL) for virtual training software is a main technical means, firstly, training process data needs to be collected, data in various types of virtual training applications are collected rapidly and efficiently, various deformation and conversion of the data can be conveniently carried out subsequently, and data easy to analyze and understand are constructed.
In situations where the data store cannot be read directly, such as when the database cannot be accessed, common collection methods include crawlers and the like.
However, the crawler can only aim at web services, and the data collection method of the virtual training program is relatively limited and cannot adapt to different application scenes.
Disclosure of Invention
The data collection system and the data collection method for the virtual training system, which are provided by the invention, are used for solving the defect that a crawler can only aim at web services in the prior art, the data collection method for the virtual training program is relatively high in limitation and cannot adapt to different application scenes, realizing automatic non-invasive data collection, and finishing the data collection into a preprocessing data collection interface convenient to call, can effectively solve the problem of data island in training application, is suitable for virtual training application and web terminals, and can support data collection of various types and be widely applied to various application scenes.
The invention provides a data collection system facing a virtual training system, which comprises: the system comprises a collection module and a preprocessing module which are in communication connection;
the collection module is used for carrying out page traversal and behavior tracking on the virtual training process of each operation terminal to obtain information data and data screenshot of each operation terminal;
and the preprocessing module is used for acquiring the preprocessing data of each operation terminal according to the information data of each operation terminal and the data screenshot.
According to the data collection system facing to the virtual training system provided by the invention, the collection module comprises: traversing the sub-module, the first acquisition sub-module and the construction sub-module;
the traversing sub-module is used for performing page traversing and user behavior tracking on the virtual training process of each running terminal based on the test framework so as to acquire control information of each operated control, and sending the control information to the acquiring sub-module;
the first obtaining submodule is specifically configured to:
under the condition that any operated control is determined to be a standard control according to the control information of any operated control, acquiring data information through any operated control;
under the condition that any operated control is determined to be a non-standard control according to the control information of any operated control, capturing an operation page to obtain a data capture;
and the construction submodule is used for constructing an index for the data information and the data screenshot of the virtual training process of each running terminal according to the training type of the virtual training.
According to the data collection system facing the virtual training system, which is provided by the invention, the collection module further comprises a tracking sub-module;
the tracking submodule is specifically configured to:
tracking user operation in the virtual training process under the condition that any operated control is determined to be a non-standard control according to control information of any operated control;
analyzing the user operation to generate a process file;
reading in the process file to store the multi-dimensional tuple;
and generating the simulation script by the multidimensional tuple.
According to the data collection system facing the virtual training system provided by the invention, the preprocessing module comprises: the device comprises a second acquisition sub-module, a character recognition sub-module and a cleaning sub-module;
the second obtaining submodule is used for obtaining the information data in the standard control according to a preset template; the preset template is generated based on the process file;
the character recognition sub-module is used for extracting character information in the data screenshot;
and the cleaning submodule is used for carrying out data cleaning on the information data and the text information so as to acquire the preprocessing data.
According to the data collection system facing the virtual training system provided by the invention, the system further comprises a storage module, the storage module is connected with the preprocessing module, and the storage module comprises: a first level storage sub-module, a second level storage sub-module and a third level storage sub-module;
the first-stage storage sub-module is used for storing the preprocessing data of all the operation terminals;
the second-stage storage sub-module is used for classifying and storing the preprocessing data of each type according to the type and index of virtual training on each running terminal;
the third-stage storage sub-module is used for converging and storing the preprocessed data classified in the second-stage storage sub-module.
According to the data collection system facing the virtual training system, which is provided by the invention, the system further comprises a management module;
the management module is respectively connected with the collection module, the preprocessing module and the storage module;
and the management module is used for carrying out state monitoring and service calling on the collecting module, the preprocessing module and the storage module.
According to the data collection system facing to the virtual training system provided by the invention, the management module comprises: the monitoring sub-module and the calling sub-module;
the monitoring submodule is used for monitoring the data collection speed of the collection module, the processing progress of the preprocessing module, and the data storage condition and the available capacity of the storage module;
and the calling sub-module is used for controlling the starting and stopping of the collecting module in the data collecting process.
The invention also provides a data collection method facing the virtual training system, which is applied to the data collection system facing the virtual training system, and comprises the following steps:
performing page traversal and behavior tracking on the virtual training process of each operation terminal to acquire information data and data screenshot of each operation terminal;
and acquiring the preprocessing data of each operation terminal according to the information data of each operation terminal and the data screenshot.
The invention also provides an electronic device, which comprises a memory, a processor and a computer program stored on the memory and capable of running on the processor, wherein the processor realizes the data collection method facing the virtual training system when executing the program.
The present invention also provides a non-transitory computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements a data collection method for a virtual training system as described in any of the above.
The invention also provides a computer program product comprising a computer program which when executed by a processor implements a data collection method for a virtual training system as described in any one of the above.
According to the data collection system and method for the virtual training system, the problem of data island in training application can be effectively solved through automatic non-invasive data collection and arrangement into the preprocessing data collection interface convenient to call, the data collection system and method for the virtual training system are suitable for virtual training application and web side, and can support data collection of various types and be widely applied to various application scenes.
Drawings
In order to more clearly illustrate the invention or the technical solutions of the prior art, the following description will briefly explain the drawings used in the embodiments or the description of the prior art, and it is obvious that the drawings in the following description are some embodiments of the invention, and other drawings can be obtained according to the drawings without inventive effort for a person skilled in the art.
FIG. 1 is a schematic diagram of a data collection system for a virtual training system according to the present invention;
FIG. 2 is a flow chart of an automated traversal process provided by the present invention;
FIG. 3 is a flow chart of the monitoring and training user operation provided by the invention;
FIG. 4 is a flow chart of the operation of tracking a specific coordinate location in a screen provided by the present invention;
FIG. 5 is a schematic diagram of a memory module according to the present invention;
FIG. 6 is a second schematic diagram of a data collection system for a virtual training system according to the present invention;
FIG. 7 is a schematic flow chart of a data collection method for a virtual training system according to the present invention;
FIG. 8 is a second flow chart of a data collection method for a virtual training system according to the present invention;
fig. 9 is a schematic structural diagram of an electronic device provided by the present invention.
Detailed Description
For the purpose of making the objects, technical solutions and advantages of the present invention more apparent, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings, and it is apparent that the described embodiments are some embodiments of the present invention, not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
At present, various virtual training software on the market is various, and how to collect training process data quickly relates to whether a group trainer can develop analysis and evaluation for training efficiency quickly. Most training systems spend a significant amount of money and time building On-line transaction (On-Line Transaction Processing, OLTP) business systems and automated training performance assessment systems associated with the training process to record various relevant data for the training process. However, most virtual training software in the market is independently developed by each company, and the software is independently closed, so that training process data cannot be directly obtained through an application interface.
The structure of web data sources is relatively clear and data is relatively easy to obtain, however, an automated data collection method is lacking, and storage and management of collected data is also lacking.
The difficulty of acquiring data for various application programs for various virtual training software and applications is higher, and the time and effort are consumed by common methods such as manual extraction and the like; the method for embedding the point in the application is only suitable for the condition of possessing the program source code and obtaining the modification authority, and has great limitation.
The following describes a data collection system and method for a virtual training system according to an embodiment of the present invention with reference to fig. 1 to 9.
Fig. 1 is a schematic structural diagram of a data collection system for a virtual training system according to the present invention, and as shown in fig. 1, the data collection system 100 includes: a collection module 110 and a preprocessing module 120 in communication;
the collecting module 110 is configured to perform page traversal and behavior tracking on a virtual training process of each operation terminal, and obtain information data and a data screenshot of each operation terminal;
the preprocessing module 120 is configured to obtain preprocessing data of each operation terminal according to the information data of each operation terminal and the data screenshot.
The virtual training process may be a service form facing virtual training, such as web service, application service, etc., and in the subsequent embodiments of the present invention, data collection of the virtual training application is taken as an example to describe the present invention, which is not considered as limiting the protection scope of the present invention.
The collection module can collect data of a plurality of operation terminals at the same time, and provides an expandable data collection service.
The page traversal is to traverse each page in the virtual training process, and the behavior tracking is to track the operation behavior of the user.
The preprocessing module performs further preprocessing such as information extraction and data cleaning on the data screenshot of the information data sent by the collecting module, so that the preprocessing data of each operation terminal can be obtained.
The data collection system facing the virtual training system provided by the invention can effectively solve the problem of data island in training application by automatically collecting non-invasive data and arranging the data collection system into the preprocessing data collection interface convenient to call, is suitable for the virtual training application and the web end, can support data collection of various types and is widely applied to various application scenes.
Optionally, the collecting module includes: traversing the sub-module, the first acquisition sub-module and the construction sub-module;
the traversing sub-module is used for performing page traversing and user behavior tracking on the virtual training process of each running terminal based on the test framework so as to acquire control information of each operated control, and sending the control information to the acquiring sub-module;
the first obtaining submodule is specifically configured to:
under the condition that any operated control is determined to be a standard control according to the control information of any operated control, acquiring data information through any operated control;
under the condition that any operated control is determined to be a non-standard control according to the control information of any operated control, capturing an operation page to obtain a data capture;
and the construction submodule is used for constructing an index for the data information and the data screenshot of the virtual training process of each running terminal according to the training type of the virtual training.
The control information includes: the control type, the control type name, the number, the position coordinates of the control and the control name.
And when the collection module collects data, collecting data of the virtual training application in a page automatic traversal and training user behavior tracking mode to obtain control information and data screenshot.
In the page automatic traversal process, for the virtual training application, a traversal submodule operates a terminal by means of a test framework (such as airtest), and performs page traversal on the virtual training application in cooperation with a depth-first traversal algorithm, and handle and control information are acquired in the traversal process.
The handle is the number of the control grabbing information, and the information in the control can be called through the handle.
Fig. 2 is a flowchart of an automated traversal process provided by the present invention, as shown in fig. 2, comprising:
firstly, extracting page elements and controls of a virtual training application through an XML path language (XML Path Language, XPath) page;
secondly, identifying the control according to the set control name and control type related to the acquired data, determining a standard control and a non-standard control, and finally performing depth-first traversal to obtain path record, redirection, popup window (installation process, permission, app prompt) and the like;
then, acquiring the handle and control information of each operated control, acquiring data information through a standard control, performing data screenshot on a non-standard control, and generating a corresponding functional process diagram;
after the path record is obtained through automatic traversal of the page, the data interface of the package can be operated by combining a test frame (such as airtest), and related operation script writing can be directly carried out;
finally, the data file builds an index optimization.
In the page traversal process, each time the user participates in training, the information in the control is triggered and the screenshot is recorded, and data collection is completed in this way. The data collection results may be indexed using a hash table or B-tree based on file name to optimize retrieval efficiency.
The automatic page traversal is suitable for common mobile terminals such as iOS, android or computer terminal applications and the like.
According to the data collection system facing the virtual training system, the page traversal and the user behavior tracking are carried out on the virtual training process, so that non-invasive data collection is realized, and the data collection system is suitable for a web end and an application end and is provided for data storage.
Optionally, the collection module further comprises a tracking sub-module;
the tracking submodule is specifically configured to:
tracking user operation in the virtual training process under the condition that any operated control is determined to be a non-standard control according to control information of any operated control;
analyzing the user operation to generate a process file;
reading in the process file to store the multi-dimensional tuple;
and generating the simulation script by the multidimensional tuple.
The simulation script is used for simulating operation on the model participating in virtual training.
Fig. 3 is a flowchart of monitoring training user operation provided by the present invention, as shown in fig. 3, including:
firstly, monitoring user operation;
secondly, generating a behavior process file;
then, generating a call relation diagram;
then, obtaining control content or screenshot through simulation;
finally, the data file builds an index optimization.
FIG. 4 is a flowchart of the operation of tracking specific coordinate positions in a screen according to the present invention, as shown in FIG. 4, including:
firstly, recording time by utilizing graphic user interface (Graphics User Interface, GUI) software, monitoring window information and specific keyboard and mouse operations (clicking, inputting, dragging and the like) performed by a participant on each control or specific coordinate position in a screen in the training process;
secondly, analyzing what controls and positions are operated by a parameter training user, what operations are respectively carried out, whether standard controls are operated, and if the standard controls are determined to be operated, directly acquiring the contents such as control attributes and the like; if the operation on the nonstandard control is determined, screenshot is carried out, the operation content and the position coordinates of a keyboard and a mouse are recorded, and a process file for referencing the detailed behaviors of the training user is generated;
finally, after the acquired process files are read in, multidimensional tuples are stored, the running sequence and circulation of the judging function of the script are judged, the sequence relation of the operation on the control and each position during simulation is extracted, the operation logic of a user is analyzed, the calling relation among the functions is obtained, and therefore a corresponding control operation directed graph is generated, and the script is output for simulation operation. The standard control is a standard program development component capable of directly acquiring values according to control attributes, and data can be directly acquired through the standard control;
and automatically simulating a behavior mode of the participant in the use process by using a simulation module, sequentially and completely operating each page and function item of the virtual training software interface, regularly acquiring an operation interface of the software user, completing screenshot of the data page and simultaneously storing information in the control.
When detecting the occurrence of a control event, monitoring the operation of a parameter training user, and acquiring a control currently operated by tools such as an aspect, a UI Automation, an airtest and the like, wherein the control information comprises: and the control type, the control type name, the number, the position coordinates of the control, the control name and the like, and collecting handle information of the operation window.
According to the data collection system facing the virtual training system, provided by the invention, the logic analysis is carried out on the behavior of the user operation, so that the simulated script is generated, and a foundation is provided for the subsequent simulation operation.
Optionally, the preprocessing module includes: the device comprises a second acquisition sub-module, a character recognition sub-module and a cleaning sub-module;
the second obtaining submodule is used for obtaining the information data in the standard control according to a preset template; the preset template is generated based on the process file;
the character recognition sub-module is used for extracting character information in the data screenshot;
and the cleaning submodule is used for carrying out data cleaning on the information data and the text information so as to acquire the preprocessing data.
In the preprocessing module, a second acquisition sub-module collects information data obtained from the control according to a preset template; the preset template is generated according to a process file formed by user behaviors;
the character recognition sub-module can extract characters in the data screenshot in a character definition and recognition mode, wherein the character recognition function firstly needs to perform preprocessing operation on an image, the area where the characters are located can be judged through image processing, and then a trained neural network model is used for character recognition.
And the cleaning sub-module performs data cleaning and data integration on the information data and the text information to obtain preprocessed data, wherein the data cleaning comprises algorithms such as dirty data removal, repeated data deduplication and the like.
According to the data collection system facing the virtual training system, the obtained data are preprocessed, so that the information density of the data is higher, and the data occupy smaller storage space.
Optionally, the system further includes a storage module, the storage module is connected to the preprocessing module, and the storage module includes: a first level storage sub-module, a second level storage sub-module and a third level storage sub-module;
the first-stage storage sub-module is used for storing the preprocessing data of all the operation terminals;
the second-stage storage sub-module is used for classifying and storing the preprocessing data of each type according to the type and index of virtual training on each running terminal;
the third-stage storage sub-module is used for converging and storing the preprocessed data classified in the second-stage storage sub-module.
In the first-stage storage sub-module, the preprocessing data after the original training process data is stored locally and the data cleaning and integration are carried out on each training data collection operation terminal is stored in one stage.
In the second-stage storage sub-module, each operation terminal of the same training type respectively collects the data stored in the corresponding first stage for unified storage;
and in the third-level storage sub-module, collecting all data in the second-level storage for unified storage.
The data storage of each stage provides a disaster recovery mechanism in the form of a main library and a standby library.
For example, in a virtual training scene, a plurality of running terminals for virtual training exist, one training type is completed on a plurality of running terminals of a class, and one-level storage of data of each running terminal is realized; because the training types among different classes are different, data of the running terminals with the same training type are required to be gathered, the data with the same training type are subjected to secondary storage, and finally, the training data of the whole training process is required to be stored, and the training data of each type in the secondary storage are gathered for tertiary storage.
FIG. 5 is a schematic diagram of a memory module according to the present invention, as shown in FIG. 5, including:
the three-level storage architecture includes: the system comprises an acquisition layer, a service layer, a resource layer and system monitoring;
in the acquisition layer, various operation terminals such as mobile equipment, PC and the like are included;
in the service layer, the system comprises a data acquisition part and a data storage part;
the data acquisition section includes: page traversal, behavior tracking, data cleansing, storage request and session management;
the data storage section includes: primary storage aiming at each running terminal, secondary storage and tertiary storage aiming at each training type;
in the resource layer, mySQL database and Hive database are included.
And the storage module is used for storing the collected data after cleaning and deduplication according to the storage rule of the three-level data storage from bottom to top.
The monitoring system respectively monitors the progress and starts and stops the control of the acquisition layer, the service layer and the resource layer.
According to the data collection system facing the virtual training system, which is provided by the invention, the multi-level storage of data is realized by a storage mode of grading multiple nodes, and the subsequent call is convenient.
Optionally, the system further comprises a management module;
the management module is respectively connected with the collection module, the preprocessing module and the storage module;
and the management module is used for carrying out state monitoring and service calling on the collecting module, the preprocessing module and the storage module.
The management module may invoke services to control the start, pause, and stop of each module operation in the overall data collection system, as well as the monitoring of the data collection process.
According to the data collection system facing the virtual training system, which is provided by the invention, other modules are managed and controlled through the management module, so that full-automatic data collection operation is realized.
Optionally, the management module includes: the monitoring sub-module and the calling sub-module;
the monitoring submodule is used for monitoring the data collection speed of the collection module, the processing progress of the preprocessing module, and the data storage condition and the available capacity of the storage module;
and the calling sub-module is used for controlling the starting and stopping of the collecting module in the data collecting process.
The management module is used for providing state monitoring and service calling for the collection module, the preprocessing module and the storage module, and can be used for monitoring real-time service conditions, data collection speed and the like of the collection module in real time, monitoring data storage conditions and capacity of the storage module in real time, starting, suspending, stopping and managing training data collection service and the like.
Fig. 6 is a second schematic structural diagram of a data collection system for a virtual training system according to the present invention, as shown in fig. 6, including: the device comprises a collection module, a preprocessing module, a storage module and a management module;
the collection module is used for automatically traversing the web or the page of the application service, tracking the user behavior and acquiring data;
the preprocessing module is used for carrying out image processing and character recognition on the image intercepted by the collecting module, and carrying out data cleaning on the data acquired by the collecting module to obtain preprocessed data;
the storage module is used for carrying out multi-stage storage, data caching and master-slave backup on the preprocessed data;
and the management module is used for user information management, data source management and acquisition task management. Wherein, the user information includes: the name, age, ID card number, etc. of the participant; the data source management is to collect and configure the data source and control the collection and management of the data according to the preset steps and logic; acquisition task management includes: the start, pause and stop of the operation of each module in the whole data collection system, and the monitoring of the data collection process.
According to the data collection system facing the virtual training system, automatic data collection is realized for each type of virtual training application program, and simultaneously, the system supports large-scale and multi-type expandable data collection and multi-level data storage management service, compared with the traditional manual method, the data collection efficiency of the virtual training application on mobile equipment such as iOS, android and the like can be saved by more than 20 times, the data collection efficiency of the virtual training application on a computer terminal can be saved by more than 10 times, and the system can support the virtual training application in mobile equipment such as iOS, android and the like and the application of the computer terminal.
The data collection method for the virtual training system provided by the invention is described below, and the data collection method for the virtual training system described below and the data collection system for the virtual training system described above can be referred to correspondingly.
According to the data collection method for the virtual training system provided by the embodiment of the invention, the execution subject can be the electronic equipment or the software or the functional module or the functional entity capable of realizing the data collection method in the electronic equipment, and the electronic equipment in the embodiment of the invention comprises but is not limited to a server. The execution body is not limited to the present invention.
Fig. 7 is a schematic flow chart of a data collection method for a virtual training system according to the present invention, as shown in fig. 7, which is applied to the data collection system for a virtual training system according to any one of the embodiments, the method includes, but is not limited to, the following steps:
firstly, in step S1, page traversal and behavior tracking are performed on a virtual training process of each running terminal, so as to obtain information data and data screenshot of each running terminal.
The virtual training process may be a service form facing virtual training, such as web service, application service, etc., and in the subsequent embodiments of the present invention, data collection of the virtual training application is taken as an example to describe the present invention, which is not considered as limiting the protection scope of the present invention.
The collection module can collect data of a plurality of operation terminals at the same time, and provides an expandable data collection service.
The page traversal is to traverse each page in the virtual training process, and the behavior tracking is to track the operation behavior of the user.
Further, in step S2, the preprocessing data of each operation terminal is obtained according to the information data of each operation terminal and the data screenshot.
The preprocessing module performs further preprocessing such as information extraction and data cleaning on the data screenshot of the information data sent by the collecting module, so that the preprocessing data of each operation terminal can be obtained.
According to the data collection method for the virtual training system, through automatic non-invasive data collection and arrangement of the preprocessing data collection interface convenient to call, the problem of data island in training application can be effectively solved, the method is applicable to virtual training application and web side, various types of data collection can be supported, and the method is widely applied to various application scenes.
Fig. 8 is a second flowchart of a data collection method for a virtual training system according to the present invention, as shown in fig. 8, including:
firstly, control information and a data screenshot are obtained in an automatic traversal or user behavior tracking mode;
secondly, extracting information data in the control according to a predefined template;
then, performing image processing on the screenshot, and performing character recognition by using a deep learning model;
furthermore, data cleaning is carried out on the information data and the identified characters;
then, carrying out data deduplication and warehousing on the cleaned data to obtain preprocessed data, and establishing an index for the preprocessed data according to the training type;
then, carrying out bottom-up multi-stage storage on the data;
and finally, monitoring the system in real time.
According to the data collection method for the virtual training system, which is provided by the invention, automatic data collection is realized for each type of virtual training application program, and simultaneously, expandable data collection supporting large scale and multiple types and multi-level data storage management service are provided, so that the data collection efficiency of the virtual training application on mobile equipment such as iOS, android and the like can be saved by more than 20 times compared with that of the traditional manual method, the data collection efficiency of the virtual training application on a computer terminal can be saved by more than 10 times, and the virtual training application in mobile equipment such as iOS, android and the like can be supported as well as the application of the computer terminal.
Fig. 9 is a schematic structural diagram of an electronic device provided by the present invention, and as shown in fig. 9, the electronic device may include: processor 910, communication interface (Communications Interface), memory 930, and communication bus 940, wherein processor 910, communication interface 920, and memory 930 communicate with each other via communication bus 940. Processor 910 may invoke logic instructions in memory 930 to perform a virtual training system oriented data collection method comprising: performing page traversal and behavior tracking on the virtual training process of each operation terminal to acquire information data and data screenshot of each operation terminal; and acquiring the preprocessing data of each operation terminal according to the information data of each operation terminal and the data screenshot.
Further, the logic instructions in the memory 930 described above may be implemented in the form of software functional units and may be stored in a computer-readable storage medium when sold or used as a stand-alone product. Based on this understanding, the technical solution of the present invention may be embodied essentially or in a part contributing to the prior art or in a part of the technical solution, in the form of a software product stored in a storage medium, comprising several instructions for causing a computer device (which may be a personal computer, a server, a network device, etc.) to perform all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a random access Memory (Random Access Memory, RAM), a magnetic disk, or an optical disk, or other various media capable of storing program codes.
In another aspect, the present invention also provides a computer program product, where the computer program product includes a computer program, where the computer program can be stored on a non-transitory computer readable storage medium, where the computer program when executed by a processor can perform a data collection method for a virtual training system provided by the foregoing methods, and the method includes: performing page traversal and behavior tracking on the virtual training process of each operation terminal to acquire information data and data screenshot of each operation terminal; and acquiring the preprocessing data of each operation terminal according to the information data of each operation terminal and the data screenshot.
In yet another aspect, the present invention further provides a non-transitory computer readable storage medium having stored thereon a computer program which, when executed by a processor, is implemented to perform the method for virtual training system-oriented data collection provided by the above methods, the method comprising: performing page traversal and behavior tracking on the virtual training process of each operation terminal to acquire information data and data screenshot of each operation terminal; and acquiring the preprocessing data of each operation terminal according to the information data of each operation terminal and the data screenshot.
The apparatus embodiments described above are merely illustrative, wherein the elements illustrated as separate elements may or may not be physically separate, and the elements shown as elements may or may not be physical elements, may be located in one place, or may be distributed over a plurality of network elements. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art will understand and implement the present invention without undue burden.
From the above description of the embodiments, it will be apparent to those skilled in the art that the embodiments may be implemented by means of software plus necessary general hardware platforms, or of course may be implemented by means of hardware. Based on this understanding, the foregoing technical solution may be embodied essentially or in a part contributing to the prior art in the form of a software product, which may be stored in a computer readable storage medium, such as ROM/RAM, a magnetic disk, an optical disk, etc., including several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the method described in the respective embodiments or some parts of the embodiments.
Finally, it should be noted that: the above embodiments are only for illustrating the technical solution of the present invention, and are not limiting; although the invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims (10)
1. A virtual training system oriented data collection system comprising: the system comprises a collection module and a preprocessing module which are in communication connection;
the collection module is used for carrying out page traversal and behavior tracking on the virtual training process of each operation terminal to obtain information data and data screenshot of each operation terminal;
and the preprocessing module is used for acquiring the preprocessing data of each operation terminal according to the information data of each operation terminal and the data screenshot.
2. The virtual training system-oriented data collection system of claim 1, wherein the collection module comprises: traversing the sub-module, the first acquisition sub-module and the construction sub-module;
the traversing sub-module is used for performing page traversing and user behavior tracking on the virtual training process of each running terminal based on the test framework so as to acquire control information of each operated control, and sending the control information to the acquiring sub-module;
the first obtaining submodule is specifically configured to:
under the condition that any operated control is determined to be a standard control according to the control information of any operated control, acquiring data information through any operated control;
under the condition that any operated control is determined to be a non-standard control according to the control information of any operated control, capturing an operation page to obtain a data capture;
and the construction submodule is used for constructing an index for the data information and the data screenshot of the virtual training process of each running terminal according to the training type of the virtual training.
3. The virtual training system oriented data collection system of claim 2, wherein the collection module further comprises a tracking sub-module;
the tracking submodule is specifically configured to:
tracking user operation in the virtual training process under the condition that any operated control is determined to be a non-standard control according to control information of any operated control;
analyzing the user operation to generate a process file;
reading in the process file to store the multi-dimensional tuple;
and generating the simulation script by the multidimensional tuple.
4. A virtual training system-oriented data collection system as claimed in claim 3, wherein the preprocessing module comprises: the device comprises a second acquisition sub-module, a character recognition sub-module and a cleaning sub-module;
the second obtaining submodule is used for obtaining the information data in the standard control according to a preset template; the preset template is generated based on the process file;
the character recognition sub-module is used for extracting character information in the data screenshot;
and the cleaning submodule is used for carrying out data cleaning on the information data and the text information so as to acquire the preprocessing data.
5. The virtual training system-oriented data collection system of any of claims 1-4, further comprising a storage module coupled to the preprocessing module, the storage module comprising: a first level storage sub-module, a second level storage sub-module and a third level storage sub-module;
the first-stage storage sub-module is used for storing the preprocessing data of all the operation terminals;
the second-stage storage sub-module is used for classifying and storing the preprocessing data of each type according to the type and index of virtual training on each running terminal;
the third-stage storage sub-module is used for converging and storing the preprocessed data classified in the second-stage storage sub-module.
6. The virtual training system-oriented data collection system of claim 5, further comprising a management module;
the management module is respectively connected with the collection module, the preprocessing module and the storage module;
and the management module is used for carrying out state monitoring and service calling on the collecting module, the preprocessing module and the storage module.
7. The virtual training system-oriented data collection system of claim 6, wherein the management module comprises: the monitoring sub-module and the calling sub-module;
the monitoring submodule is used for monitoring the data collection speed of the collection module, the processing progress of the preprocessing module, and the data storage condition and the available capacity of the storage module;
and the calling sub-module is used for controlling the starting and stopping of the collecting module in the data collecting process.
8. A data collection method for a virtual training system, applied to the data collection system for a virtual training system according to any one of claims 1 to 7, the method comprising:
performing page traversal and behavior tracking on the virtual training process of each operation terminal to acquire information data and data screenshot of each operation terminal;
and acquiring the preprocessing data of each operation terminal according to the information data of each operation terminal and the data screenshot.
9. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the virtual training system-oriented data collection method of claim 8 when the program is executed by the processor.
10. A non-transitory computer readable storage medium having stored thereon a computer program, which when executed by a processor implements the virtual training system-oriented data collection method of claim 8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310752938.8A CN116991659A (en) | 2023-06-25 | 2023-06-25 | Virtual training system-oriented data collection system and method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310752938.8A CN116991659A (en) | 2023-06-25 | 2023-06-25 | Virtual training system-oriented data collection system and method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116991659A true CN116991659A (en) | 2023-11-03 |
Family
ID=88530981
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310752938.8A Pending CN116991659A (en) | 2023-06-25 | 2023-06-25 | Virtual training system-oriented data collection system and method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116991659A (en) |
-
2023
- 2023-06-25 CN CN202310752938.8A patent/CN116991659A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| EP3798846B1 (en) | Operation and maintenance system and method | |
| CN110428127B (en) | Automatic analysis method, user equipment, storage medium and device | |
| US20190108223A1 (en) | Automated system data migration | |
| CN112507623B (en) | Method and system for constructing algorithm middle station | |
| CN106227654A (en) | A kind of test platform | |
| CN109242431B (en) | Enterprise management method and system based on data system | |
| CN115237857A (en) | Log processing method and device, computer equipment and storage medium | |
| CN114419631A (en) | Network management virtual system based on RPA | |
| CN113051161A (en) | API misuse detection method based on historical code change information | |
| CN117792882A (en) | Communication network fault log analysis method based on large language model assistance | |
| CN112463801A (en) | Report form pushing method and device, terminal equipment and storage medium | |
| CN116049115B (en) | Software log processing method | |
| CN112650796A (en) | Automatic application data collection and storage management system | |
| CN109146306B (en) | Enterprise management system | |
| CN116991659A (en) | Virtual training system-oriented data collection system and method | |
| CN116302984A (en) | Root cause analysis method and device for test task and related equipment | |
| CN116069628A (en) | Intelligent-treatment software automatic regression testing method, system and equipment | |
| CN115905016A (en) | BIOS Setup search function test method and device, electronic equipment and storage medium | |
| CN112667617A (en) | Visual data cleaning system and method based on natural language | |
| CN113887913A (en) | Efficiency evaluation method and device combining RPA and AI and electronic equipment | |
| CN111598159B (en) | Training method, device, equipment and storage medium of machine learning model | |
| CN115203057B (en) | Low code test automation method, device, equipment and storage medium | |
| CN114490315B (en) | Automatic testing method and device for Web application program | |
| US11507728B2 (en) | Click to document | |
| CN116755830A (en) | GPT-based container escape detection method, device and equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |