CN113806331A

CN113806331A - Data processing method and device

Info

Publication number: CN113806331A
Application number: CN202110929532.3A
Authority: CN
Inventors: 于海涛; 陈长城; 张宗禹; 王廓
Original assignee: Alibaba China Co Ltd; Alibaba Cloud Computing Ltd
Current assignee: Alibaba China Co Ltd; Alibaba Cloud Computing Ltd
Priority date: 2021-08-13
Filing date: 2021-08-13
Publication date: 2021-12-17

Abstract

The present specification provides a data processing method and a data processing apparatus, wherein the data processing method is applied to a data processing system, the system includes a data acquisition apparatus applied to a target server and a data collection apparatus applied to a data processing terminal different from the target server, wherein the data acquisition apparatus acquires initial operation data and corresponding data content of the target application, determines target operation data based on the data content, and sends the target operation data to the data collection apparatus; and the data collection device performs syntax analysis on the received target operation data, determines characteristic parameters in the target operation data, and processes the target operation data based on the characteristic parameters to obtain the target data.

Description

Data processing method and device

Technical Field

The present disclosure relates to the field of data processing technologies, and in particular, to a data processing method. The present description also relates to a data processing system, a computing device, a computer readable storage medium, and a computer program.

Background

With the continuous development of science and technology, many enterprises begin to migrate the existing database to the cloud database, so that digital transformation or industrial upgrading is realized, but in the database migration process, due to the structural difference between the existing database and the cloud database, the difference between the grammatical characteristics of the existing database and the cloud database, partial incompatibility between database objects and application codes and other problems, error reporting or inconsistent semantics of database execution are caused; in order to ensure that the database runs normally after being replaced, execution statements (such as SQL statements) generated by the application need to be collected, and incompatible points in the database and the application need to be found based on the SQL statements.

However, in the prior art, a database periodic collection mode is adopted to directly monitor the database, so as to collect an SQL list of the database, but in this way, on one hand, since the database generates SQL statements during the operation process, it is impossible to distinguish whether the SQL statements collected from the database are initiated by an application. On the other hand, in the process of monitoring the database, the performance of the database and the application is greatly influenced, and the running efficiency of the database and the application is reduced.

Disclosure of Invention

In view of this, the embodiments of the present specification provide a data processing method. The present specification also relates to a data processing system, a computing device, a computer readable storage medium, and a computer program to address technical deficiencies in the art.

According to a first aspect of the embodiments of the present specification, there is provided a data processing method applied to a data processing system, the system including a data acquisition device applied to a target server and a data collection device applied to a data processing terminal different from the target server, wherein,

the data acquisition device acquires initial operation data and corresponding data content of a target application, determines target operation data based on the data content, and sends the target operation data to the data collection device;

and the data collection device performs syntax analysis on the received target operation data, determines characteristic parameters in the target operation data, and processes the target operation data based on the characteristic parameters to obtain the target data.

According to a second aspect of the embodiments of the present specification, there is provided a data processing system including a data acquisition device applied to a target server and a data collection device applied to a data processing side different from the target server, wherein,

the data acquisition device is configured to acquire initial operation data and corresponding data content of a target application, determine target operation data based on the data content, and send the target operation data to the data collection device;

the data collection device is configured to parse the received target operation data, determine a characteristic parameter in the target operation data, and process the target operation data based on the characteristic parameter to obtain target data.

According to a third aspect of embodiments herein, there is provided a computing device comprising:

a memory and a processor;

the memory is for storing computer-executable instructions, and the processor is for executing the computer-executable instructions, which when executed by the processor, implement the steps of any of the data processing methods.

According to a fourth aspect of embodiments herein, there is provided a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of any of the data processing methods.

According to a fifth aspect of embodiments herein, there is provided a computer program, wherein the computer program, when executed in a computer, causes the computer to perform the steps of any of the data processing methods.

The data processing method provided by the specification comprises a data acquisition device applied to a target server and a data acquisition device applied to a data processing end different from the target server, wherein the data acquisition device is configured to acquire initial operation data and corresponding data content of a target application, determine target operation data based on the data content, and send the target operation data to the data acquisition device; the data collection device is configured to parse the received target operation data, determine a characteristic parameter in the target operation data, and process the target operation data based on the characteristic parameter to obtain target data.

Specifically, the data processing method obtains target operation data based on data content of initial operation data generated in an operation process of a target application in a target server through a data acquisition device applied to the target server, effectively avoids the problem that whether the execution statement acquired from a database cannot be distinguished is an execution statement initiated by the application, can accurately and quickly obtain the execution statement sent by the application, and performs heterogeneous processing on the target operation data through a data collection device applied to a data processing end different from the target server after obtaining the target operation data to obtain the target data; therefore, the influence on the application and the database is reduced, and the running efficiency of the application and the database is effectively improved.

Drawings

Fig. 1 is a flowchart of a data processing method provided in an embodiment of the present specification;

fig. 2 is a flowchart illustrating a data processing method applied in a scenario of acquiring running data of an application according to an embodiment of the present specification;

FIG. 3 is a block diagram of a data processing system according to an embodiment of the present disclosure;

fig. 4 is a block diagram of a computing device according to an embodiment of the present disclosure.

Detailed Description

In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present description. This description may be embodied in many different forms and should not be construed as limited to the embodiments set forth herein, as those skilled in the art will be able to make and use the present disclosure without departing from the spirit and scope of the present disclosure.

The terminology used in the description of the one or more embodiments is for the purpose of describing the particular embodiments only and is not intended to be limiting of the description of the one or more embodiments. As used in one or more embodiments of the present specification and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the term "and/or" as used in one or more embodiments of the present specification refers to and encompasses any and all possible combinations of one or more of the associated listed items.

It will be understood that, although the terms first, second, etc. may be used herein in one or more embodiments to describe various information, these information should not be limited by these terms. These terms are only used to distinguish one type of information from another. For example, a first can also be referred to as a second and, similarly, a second can also be referred to as a first without departing from the scope of one or more embodiments of the present description. The word "if" as used herein may be interpreted as "at … …" or "when … …" or "in response to a determination", depending on the context.

First, the noun terms to which one or more embodiments of the present specification relate are explained.

A database: a repository of data is organized, stored, and managed by a data structure.

ADAM Database and Application Migration (Advanced Database & Application Migration), referred to as ADAM for short, is a product used to help heterogeneous databases and applications evaluate and structure migrate a full link. Databases and applications can be migrated to cloud databases (public or private clouds) through ADAMs, significantly reducing the technical difficulty and cost of going to the cloud. The ADAM can comprehensively evaluate cloud availability, cost and cloud storage type selection, tools for implementation assistance, data and application migration are built in, and reliable and rapid cloud availability is guaranteed.

Dynamic collection: data collection is performed while the application service is running in real time.

A source library: a database of customer scripts.

Target library: the target database to which the customer desires to migrate.

Heterogeneous database migration: the customer migrates from the native database to the target database.

Syntax incompatibility: the original database writing method cannot be compatible on a new target database due to different grammatical characteristics, and is represented as an execution error report or a semantic error.

Sensitive information: all information that is not properly used, is not authorized to be contacted by a person, or is not conducive to the privacy rights enjoyed by the user.

Desensitization: and the data (sensitive information) which is possibly sensitive is replaced and deleted by special characters, so that the output content is ensured not to contain the actual data and query conditions of the client.

APM: application Performance Management (APM) is simply referred to as APM.

SQL: structured Query Language (SQL) is simply referred to.

JVM: refers to a Java virtual Machine (Java virtual Machine).

Java: is a door-to-object programming language.

With the development of digital transformation and technology, currently, many enterprises are undergoing digital transformation or industrial upgrading, and it is desired to migrate the original database of the enterprise to a cloud database, and heterogeneous database migration will certainly involve application modification. The syntax, semantics and usage of the original database and the cloud database of an enterprise are different, and the original database object and application code are partially incompatible with the cloud database, so that the execution error report or semantics of the database are inconsistent. Therefore, in order to ensure that the database runs normally after being replaced, incompatible points of the database and the application need to be found and corrected, so that the application is modified, which is the most time-consuming place in the migration process of the heterogeneous database.

And the application of SQL is formed by particularly dispersing and splicing multiple scenes, so that all SQL is difficult to be combed out completely. Based on the above problems, the places to be modified can be gradually checked by manually writing test cases, the test verification of one function and one function is performed, and then the gradual modification is performed by finding a modification mode, which causes a very long modification period and consumes much labor and time.

Based on the above problems, one way is to acquire the SQL list by directly monitoring the database for acquisition, which, although it is possible to acquire SQL, cannot distinguish whether the acquired SQL is initiated by an application, which application the acquired SQL is sent by, and cannot locate the specific location of the acquired SQL in the application.

Another way is to obtain the running data of the application through the APM tool, but this way is only applied to help the client monitor the health of the application, has a great influence on the performance of the application, and cannot perform desensitization processing on the acquired SQL.

In view of this, in the present specification, a data processing method is provided, and the present specification simultaneously relates to a data processing system, a computing device, and a computer-readable storage medium, which are described in detail one by one in the following embodiments.

Referring to fig. 1, fig. 1 shows a flowchart of a data processing method provided according to an embodiment of the present specification, where the data processing method is applied to a data processing system, where the system includes a data acquisition device applied to a target server and a data collection device applied to a data processing end different from the target server, and specifically includes the following steps.

Step 102: the data acquisition device acquires initial operation data and corresponding data content of a target application, determines target operation data based on the data content, and sends the target operation data to the data collection device.

In practical application, the data processing system can be an application dynamic collector deployed on ADAM, and the application dynamic collector can help a client to comb out all real SQL, call stack position and other operation information at one time without modifying application codes, and perform data desensitization and information processing on the operation information to provide basic data for workload evaluation of subsequent heterogeneous database migration. The application dynamic collector comprises two models: agent module and collector module; the agent module is also called a dynamic collector agent, the dynamic collector agent is deployed in the target server and is used for collecting the running data of the application program in the target server, and in practical application, the dynamic collector agent comprises at least one; the collector module is also called as a dynamic collector, the dynamic collector is deployed at a data processing end different from the target server, and is used for processing the acquired target operation data, and the target server can be an application server; the data processing side different from the target server includes, but is not limited to, a client side and a server different from the application server.

The data acquisition device can be understood as a device for acquiring target operation data of the target application in a periodic acquisition mode; the data acquisition devices are deployed on the application server, and at least one of the data acquisition devices is included; in practical application, the data acquisition device can be a dynamic collector agent applying a dynamic collector, and can dynamically acquire target operation data of target application in an application server by starting the collector agent; in the embodiment of the present specification, two ways of starting the collector agent can be provided according to the actual operation condition of the client application:

under the condition that the client application is stable, the collector agent can be started by inputting a command;

under the condition that the client application is frequently started/closed, a mode of starting the collector agent by inputting a command can cause a client to need frequent command input, and on the basis, in the process of starting the application by the client, a parameter is added in an application starting command document and the collector agent is started by the parameter, so that the collector agent can be started or stopped along with the client application in the process of starting or stopping the client application.

The data collection device may be understood as a device for performing heterogeneous processing on the received target operation data, and in practical applications, the data collection device may be a dynamic collector of an application dynamic collector; the dynamic collector can be started by inputting a command, and the received target operation data can be subjected to heterogeneous processing by starting the dynamic collector, so that the target data is obtained.

The target application can be understood as an application needing to be modified in the migration process of the heterogeneous database; in practical application, because the source library and the target library have an incompatibility problem, an incompatibility point in the application needs to be corrected, so that the normal operation of the replaced database is ensured.

The initial operating data can be understood as all operating data generated by the target application during operation.

The data content may be understood as data identification, parameter information, functions, etc. of the initial operation data.

The target operation data can be understood as specific operation data generated in the actual operation process of the target application, and the target operation data includes, but is not limited to, URL information, SQL statements, call stack information, SQL-dbinfo information, and SQL response information.

Specifically, the data acquisition device applied to the target server acquires initial operation data of a plurality of target applications in a periodic acquisition mode and determines data content corresponding to the initial operation data; one or more target operation data are determined from the initial operation data based on the data content, and the one or more target operation data are sent to a data collection device.

For example, in the case of using a data processing system as an application dynamic collector, the data processing system is further described in detail, wherein the target operation data may be an SQL statement.

The application dynamic collector comprises two parts: a dynamic collector agent and a dynamic collector; aiming at the dynamic collector agent, adding the execution logic of the dynamic collector agent into JVMs of a plurality of client applications through a bytecode enhancement technology and starting the dynamic collector agent; the started dynamic collector agent can monitor the main link of the client application in real time, dynamically collect the running data of the plurality of client applications in a periodic collection mode, and obtain SQL sentences generated by the plurality of client applications from the running data through the data content corresponding to the running data.

The dynamic collector agent records the collected SQL statements into an asynchronous thread pool on a main link without performing any processing on the SQL statements; and sending SQL statements in the asynchronous thread pool to a dynamic collector for processing in an asynchronous sending mode through a processing thread.

In practical application, the application dynamic collector has rich self-monitoring mechanisms, automatically suspends work when the client application is in a running peak, and performs peak staggering work after the running peak is reduced.

In the embodiment of the specification, the target operation data acquired by the dynamic acquisition unit includes call stack information, and by acquiring the call stack information, the specific position of the application code for calling the SQL is reversely recorded when the SQL is acquired, so that the SQL is conveniently and directly positioned at the specific position of the application code for calling the SQL when the client application is subsequently modified, and the modification efficiency is improved.

Furthermore, the data generated by the target application during the operation process is diverse, which results in that the data acquisition device needs to acquire all the operation data generated by the target application during the process of acquiring the target operation data, detect, identify and determine the data content of the acquired operation data, and acquire the target operation data based on the data content, but in the process of acquiring the target operation data from the initial operation data by the data acquisition device based on the data content, because the initial operation data is diverse in types and large in quantity, the efficiency of acquiring the target operation data from the initial operation data based on the data content is low, and a large amount of computer processing resources are wasted, in order to solve the above problems, an embodiment of the present specification acquires the target operation data by determining the data format of the initial operation data, the specific implementation is as follows.

The data acquisition device determines target operating data based on the data content, including:

the data acquisition device determines a data format of the initial operation data based on the data content, and acquires target operation data from the initial operation data based on the data format.

The data format may be understood as a format of the running data generated by the target application, and in actual applications, the types of the running data generated by the target application are various, such as URL data and SQL data. Each type of operational data has a particular data format.

Specifically, after determining the data content in the initial operating data, the data acquisition device determines the data format of the initial operating data based on the data content, screens out the initial operating data conforming to the target data format from the initial operating data based on the data format, and determines the initial operating data conforming to the target data format as the target operating data, thereby completing the acquisition of the target operating data.

Following the above example, taking a scenario in which a data processing method is applied to acquiring SQL generated by an application as an example, details of acquiring target operation data from initial operation data based on a data format are described.

After determining the data content of all the operation data, the dynamic collector agent determines the data format of each operation data based on the data content, and matches the data format of each operation data with a target data format, where the target data format may be an SQL statement format.

Under the condition that the data format of the operating data is consistent with the SQL statement format, determining the operating data as the operating data to be acquired, thereby completing the acquisition of the target operating data;

and under the condition that the data format of the operating data is inconsistent with the SQL statement format, indicating that the operating data is not the operating data needing to be acquired, and discarding the operating data.

In the embodiment of the description, the data acquisition device determines the data format of the initial operation data based on the data content, and acquires the target operation data from the initial operation data based on the data format. Therefore, the target operation data is efficiently acquired, the problem that the efficiency of acquiring the target operation data from the initial operation data is low is solved, and the accuracy of the target operation data is effectively improved.

Further, in another case of the embodiment of the present specification, the data content includes a data identifier;

accordingly, the data acquisition device determines target operation data based on the data content, including:

and the data acquisition device determines the initial operation data as target operation data under the condition that the data identification of the initial operation data is consistent with the preset data identification.

The data identifiers of the initial operating data can be understood as identifiers for characterizing the initial operating data, each data identifier uniquely characterizing one identifier of the initial operating data.

The preset data identifier may be set according to different actual application scenarios, and in the embodiment of the present specification, the preset data identifier is not specifically limited, for example, the preset data identifier may be an identifier of an SQL statement.

Specifically, the data acquisition device matches the data identifier of the initial operating data with a preset data identifier after determining the data identifier of the initial operating data, and determines the initial operating data as the target operating data under the condition that the data identifier of the initial operating data is consistent with the preset data identifier.

Following the above example, the determination of the target operational data by the data identification of the initial operational data is described in further detail. The running data may include SQL statements, URL information, http information, etc. generated by the client application during running.

After obtaining the running data generated by the client application, the dynamic collector agent determines a data identifier of the running data, matches the data identifier with an identifier of an SQL statement, and determines the running data corresponding to the data identifier as the SQL statement when the data identifier is consistent with the identifier of the SQL statement.

In the embodiment of the specification, the initial operation data is determined as the target operation data under the condition that the data identifier of the initial operation data is consistent with the preset data identifier, so that the target operation data is efficiently acquired, the problem that the efficiency of acquiring the target operation data from the initial operation data is low is solved, the accuracy of the target operation data is effectively improved, and the precision of the target data is further improved.

Step 104: and the data collection device performs syntax analysis on the received target operation data, determines characteristic parameters in the target operation data, and processes the target operation data based on the characteristic parameters to obtain the target data.

The characteristic parameter can be understood as sensitive information contained in the target operation data. In practical applications, the characteristic parameters may be set according to practical applications, and this is not limited in this embodiment of the present specification. For example, in the case where the target operation data is SQL information, the characteristic parameter may be "table _ name", "username ═ 139 ×," passprod ═ 123 × "; for example, in the case where the target operation data is a URL statement, the feature parameter may be "iPhone ═ 182 ×".

Processing the target operation data includes but is not limited to desensitizing or de-duplicating the target operation data; correspondingly, the target data can be understood as the processed target operation data.

Specifically, after receiving target operation data sent by a data acquisition device, a data acquisition device performs syntax analysis on the target operation data through a syntax parser deployed in the data acquisition device, so as to obtain characteristic parameters in the target operation data, and the target operation data is processed based on the characteristic parameters, so as to obtain the target data.

Following the above example, the feature parameters obtained by parsing the target operation data are further described in detail.

After decompressing the dynamic collector, the dynamic collector is started through a command, and the started dynamic collector can receive the SQL statement sent by the dynamic collector agent.

After receiving the SQL statement sent by the dynamic collector agent, the dynamic collector agent inputs the SQL statement into a syntax parser deployed on the dynamic collector agent, and the SQL statement can be parsed by the syntax parser, so that sensitive information in the SQL statement is obtained, and the SQL statement is desensitized, so that the SQL statement not containing any application sensitive information is obtained.

In the embodiment of the present description, the data collection device performs syntax analysis on the received target operation data, so as to quickly and accurately obtain the characteristic parameters in the target operation data, improve the efficiency of obtaining the target object, and save computer processing resources.

Further, in the embodiment of the specification, the data collection device processes the target operation data based on the characteristic parameter, so as to obtain the target data, and meanwhile, the target data can also be obtained through the characteristic parameter and the identification information corresponding to the characteristic parameter, and a specific implementation manner is as follows.

The data collection device processes the target operation data based on the characteristic parameters to obtain target data, and the data collection device comprises:

and the data collection device determines identification information corresponding to the characteristic parameters, and processes the target operation data based on the characteristic parameters and the identification information corresponding to the characteristic parameters to obtain target data.

In this embodiment of the present description, different pieces of feature information correspond to different pieces of feature information, for example, in a case that the feature information is "username ═ 139 ×", the piece of feature information corresponding to the feature parameter may be a user name; the characteristic information is 'iPhone 182 x', and the identification information corresponding to the characteristic parameter is a mobile phone number.

Specifically, after determining the characteristic parameters in the target operation data, the data collection device determines the identification information corresponding to the characteristic parameters, and processes the target operation data based on the characteristic parameters and the identification information corresponding to the characteristic parameters, thereby obtaining the target data.

Following the above example, the processing of the target operation data based on the characteristic parameters and the identification information corresponding to the characteristic parameters is further described in detail.

After the characteristic parameters in the SQL statement are determined by the syntax parser, the dynamic collector identifies the identification information corresponding to each characteristic parameter based on the syntax parser.

And the dynamic collector processes the SQL statement based on the characteristic parameters and the identification information corresponding to the characteristic parameters, so as to obtain the target SQL statement.

Further, the processing, by the data collection device, the target operation data based on the characteristic parameter and the identification information to obtain target data includes:

the data collection device processes the target operation data based on the characteristic parameters and the identification information to obtain preprocessed data;

and the data collection device performs duplicate removal processing on the preprocessed data to obtain target data.

In the case that the target operation data is an SQL statement, the preprocessed data may be understood as the desensitized SQL statement.

Specifically, after determining identification information corresponding to the characteristic parameters, the data collection device processes the target operation data based on the characteristic parameters and the identification information to obtain preprocessed data; and the target data is obtained by performing deduplication processing on the preprocessed data.

Following the above example, the target data obtained by obtaining the preprocessed data through the characteristic parameters and the identification information and performing deduplication processing on the preprocessed data will be further described in detail.

After determining the characteristic parameters and the identification information corresponding to the characteristic parameters based on the parser, the dynamic collector determines whether the identification information meets a preset identification condition, where the preset identification condition may be whether the identification information is target identification information, and the target identification information may be a user name, a password, and a mobile phone number. And under the condition that the identification information is consistent with the preset target identification, the identification information meets the preset identification condition, the characteristic parameter corresponding to the identification information is determined as a sensitive characteristic parameter needing desensitization, and desensitization is carried out on the sensitive characteristic parameter to obtain the desensitized SQL statement.

And after the dynamic collector acquires the desensitized SQL statement, performing deduplication processing on the desensitized SQL statement, thereby acquiring a target SQL statement.

In the embodiment of the specification, the data collection device obtains preprocessing data based on the characteristic parameters and the identification information; and target data is obtained by carrying out deduplication processing on the preprocessed data. The method and the device realize that the privacy information of the client is prevented from being revealed through desensitization processing, avoid obtaining a large amount of repeated data through deduplication processing, save computer processing resources and improve the subsequent database migration efficiency.

Further, the data collecting device processes the target operation data based on the characteristic parameters and the identification information to obtain preprocessed data, including:

the data collection device determines the characteristic parameters corresponding to the identification information as target characteristic parameters under the condition that the identification information corresponding to the characteristic parameters meets preset identification conditions;

and the data collection device replaces the target characteristic parameters in the target operation data based on preset target characters to obtain preprocessed data.

The preset identification condition may be set according to actual application, and this is not limited in this description embodiment. For example, the preset identification condition may be whether the identification information is the target identification information; the target identification information may be a user name, a password, and a mobile phone number.

The preset target character may be set according to practical applications, and this is not limited in this embodiment of the specification. For example, the special characters include words and/or symbols.

Specifically, the data collection device determines the characteristic parameter corresponding to the identification information as a target characteristic parameter under the condition that the identification information corresponding to the characteristic parameter meets a preset identification condition; and replacing the target characteristic parameters in the target operation data by preset target characters so as to obtain preprocessed data.

Following the above example, the determined target feature parameters are replaced based on the preset target characters, so as to obtain the preprocessed data, which is further described in detail.

After determining the characteristic parameters and the identification information corresponding to the characteristic parameters based on the syntax parser, the dynamic collector judges whether the identification information meets preset identification conditions, whether the preset identification conditions can be that the identification information is target identification information, and under the condition that the identification information is consistent with the preset target identification, the identification information meets the preset identification conditions, determines the characteristic parameters corresponding to the identification information as sensitive characteristic parameters needing desensitization, and replaces the sensitive characteristic parameters needing desensitization through preset target characters, so that desensitization operation on SQL statements is completed, and the desensitized SQL statements are obtained.

In the embodiment of the present specification, the data collection device determines, as a target feature parameter, the feature parameter corresponding to the identification information when the identification information corresponding to the feature parameter satisfies a preset identification condition; and replacing the target characteristic parameters in the target operation data by preset target characters so as to obtain preprocessed data. Therefore, desensitization processing is effectively carried out on the target operation data, the data security of client application is guaranteed, and the problem of data leakage is avoided.

In this embodiment of the present specification, the obtained target operation data in multiple formats may also be adjusted into a uniform format, and the adjusted target operation data is stored in an application database of a client, so that the client can process the target operation data conveniently, and a specific implementation manner is as follows.

After the data collection device determines the characteristic parameters in the target operation data, the data collection device further comprises:

and the data collection device carries out format processing on the target operation data to obtain processed target operation data, and stores the processed target operation data to an application database.

The application database may be understood as a database for storing client application information, and in practical applications, the application database is a local database of a client.

Specifically, the data collection device adjusts the acquired target operation data into a uniform format, acquires processed target operation data, and stores the processed target operation data in the application database.

Following the above example, the details of obtaining the processed target operation data and storing the processed target operation data in the application database will be further described.

After acquiring the SQL statements of the multiple client applications, the dynamic collector converts the data formats of the SQL statements of the multiple client applications into a uniform data format by performing format processing on the SQL statements, and stores the converted SQL statements in a local database of a client.

In this embodiment of the present specification, the data collection device performs formatting processing on the acquired target operation data, and stores the processed target operation data in the application database. Therefore, the client can process the target operation data conveniently, and the use experience of the client is improved.

In order to obtain the running state information of the data acquisition device and the data collection device in real time and facilitate the monitoring of the data acquisition device and the data collection device by a user, in the embodiment of the present specification, the state data of the data acquisition device and the data collection device is determined according to the attribute information of the target data, so as to obtain the running states of the data acquisition device and the data collection device in real time, and the specific implementation manner is as follows.

The data collection device processes the target operation data based on the characteristic parameters, and after the target data is obtained, the data collection device further comprises:

the data collection device obtains attribute information of the target data, and determines the data collection device and state data of the data collection device based on the attribute information.

The attribute information of the target data may be set according to different practical applications, which is not limited in this embodiment of the specification. For example, the attribute information may be information such as the number of target data, the generation time of the target data, and the like.

The state data can be understood as a dynamic collector and the collection time of the dynamic collector, the collection times and the average collection time in a preset time period, and the success rate of the dynamic collector.

Specifically, the data collection device determines attribute information of the target data, thereby determining status data indicating the operation states of the data collection device and the data collection device based on the attribute information.

Following the above example, taking the scenario that the data processing method is applied to acquiring the SQL generated by the application as an example, the determination of the status data of the data acquisition device and the data collection device is further described in detail.

After the dynamic collector obtains the target SQL sentences, the attribute information such as the number, the generation time and the like of the target SQL sentences is determined, and the running states of the dynamic collector and the dynamic collector agent are obtained based on the attribute information, so that a user can conveniently monitor the running states of the dynamic collector and the dynamic collector agent.

In the embodiment of the specification, the attribute information of the target data is acquired by the data collection device, the state data of the data collection device and the data collection device is determined based on the attribute information, the running state information of the data collection device and the data collection device can be accurately acquired, and a user can conveniently monitor the data collection device and the data collection device.

In an embodiment of the present specification, after obtaining the target data and the status data of the data acquisition device and the data collection device by the data acquisition device and the data collection device, the data processing method further includes:

and the data collection device uploads the target data and the state data to a target database, so that compatibility detection of the target data in the target database is realized.

The target database may be understood as a database to which the client application needs to be migrated during the migration process of the heterogeneous database, such as a cloud database.

Specifically, the data collection device uploads the target data and the state data to the target database, so that compatibility detection of the target data in the target database is realized.

Following the above example, the compatibility check of the target data in the target database will be described in further detail.

After the dynamic collector uploads the acquired target SQL statements and the state information to the cloud database, compatibility detection of the target SQL in the target database can be achieved based on an application evaluation platform, and therefore evaluation of workload of heterogeneous database migration is judged.

In the embodiment of the specification, the target data and the state data are uploaded to the target database through the data collection device, compatibility detection of the target data in the target database is achieved, framework topology analysis of a client application cluster is facilitated, and therefore workload of heterogeneous database migration is effectively evaluated.

In an embodiment of this specification, the data processing method further includes:

the data acquisition device acquires dynamic operation information of a target application and sends the dynamic operation information to the data acquisition device;

and the data collection device processes the received dynamic operation information and sends the processed dynamic operation information to a target database, so that the performance detection of the target application in the target database is realized.

The dynamic operation information may be understood as performance information of the target application in the operation process, such as application server disk information, JVM performance information, and application server memory state information.

Specifically, the data acquisition device acquires dynamic operation information of the target application and sends the dynamic operation information to the data acquisition device; and after receiving the dynamic operation information, the data collection device processes the dynamic operation information and sends the processed dynamic operation information to the target database, so that the performance detection of the target application in the target database is realized.

Following the above example, further description is made on performance detection of the target application in the target database by acquiring the dynamic operation information of the target application and sending the processed dynamic operation information to the target database.

The dynamic collector agent acquires dynamic running information of the client application in the running process in a periodic collection mode, and sends the running dynamic information to the dynamic collector for processing in an asynchronous sending mode.

After receiving the dynamic operation information, the dynamic collector processes the format of the dynamic operation information, converts the dynamic operation information of the plurality of client applications into dynamic operation information with a uniform format, and sends the processed dynamic operation information to the cloud database for storage.

After the dynamic operation information is sent to the cloud database, performance detection of the target application in the target database is achieved through the application evaluation platform, and therefore performance of the target application is determined.

In the embodiment of the specification, the dynamic running information of the target application is acquired through the data acquisition device, the dynamic running information is processed through the data acquisition device, and the processed dynamic running information is sent to the target database, so that the performance detection of the target application in the target database is realized, the performance condition of the target application is accurately determined, and the use experience of a client is improved.

In the data processing method provided in the embodiment of the present specification, the target operation data of the target application is acquired by the data acquisition device applied to the target server, so that the problem that whether the acquired SQL is initiated by the application cannot be distinguished is effectively avoided, the SQL sent by the application can be accurately and quickly acquired, and the target operation data is subjected to heterogeneous processing by the data collection device applied to the data processing terminal different from the target server, so as to acquire the target data; the influence on the application and the database is reduced, and the running efficiency of the application and the database is effectively improved.

Meanwhile, by adopting a heterogeneous processing scheme of mutually separating data acquisition and data processing, the method has almost no influence on a request link, an application memory and a CPU; and desensitizing the acquired target operation data, thereby ensuring that the output target data does not contain any sensitive information.

The following description will further describe the data processing method with reference to fig. 2 by taking an application of the data processing method provided in this specification in a scenario of acquiring running data of the application as an example. Fig. 2 is a schematic flowchart of a data processing method applied in a scenario of acquiring running data of an application according to an embodiment of the present specification, and based on the schematic flowchart, it can be seen that the data processing method applied in the scenario of acquiring running data of an application according to the present specification includes two devices: dynamic collectors and dynamic collectors.

The dynamic collector is deployed on the application server and can collect operation information generated in the operation process of the client application and system dynamic information of the client application in a periodic collection mode; the client application may be deployed in multiple application servers. After the dynamic collector obtains the operation information and the system dynamic information, the collected information is sent to the dynamic collector for processing in an asynchronous sending mode.

The dynamic collector is deployed on a data processing end different from the application server, after receiving information sent by the dynamic collector, the dynamic collector performs semantic analysis on the operation information of the client application through a grammar parser for the operation information generated in the operation process of the client application, identifies data such as parameters and functions in the operation information, and replaces data which may be sensitive by using special characters to realize desensitization processing on the operation information, so that the operation data without sensitive information is obtained; after the operation data without sensitive information is obtained, the operation data is subjected to duplicate removal processing, and then the operation data subjected to duplicate removal processing is sent to the cloud database, so that the workload of migration of the heterogeneous database is evaluated through the operation data in the cloud database.

And for the system dynamic information of the client application, the dynamic collector carries out standardized processing on the system dynamic information, converts the system dynamic information into a uniform data format, and sends the processed system dynamic information to the cloud database, so that the performance detection of the client application is realized through the system dynamic information in the cloud data.

Based on the above description, the data processing method applied in the scenario of acquiring the running data of the application provided in this specification specifically includes the following steps:

step 220: and the dynamic collector acquires the running information generated by the client application in the server in the running process in a periodic collection mode and sends the running information to the dynamic collector.

The dynamic collector may be understood as the data collecting device in the above embodiments.

The operation information may be understood as target operation data in the above embodiments, and the operation information may be URL information, SQL-dbinfo information, SQL response information, and call stack information.

The dynamic collector may be understood as the data collection device in the above-described embodiment.

The dynamic collector can process the operation information after receiving the operation information sent by the dynamic collector, and sends the processed operation information to the cloud database and the local database, and the specific implementation steps comprise:

step 2202: and the dynamic collector analyzes the syntax of the received operation information through the syntax analyzer and sends the analyzed operation information to the formatting processing module for processing.

Step 2204: and the formatting processing module carries out formatting processing on the received operation information and sends the processed operation information to a local database for storage.

The local database may be understood as the application database in the above embodiments.

Step 2206: and the dynamic collector analyzes the syntax of the received operation information through the syntax analyzer and sends the analyzed operation information to the data desensitization module for desensitization.

Step 2208: and the data desensitization module performs desensitization processing on the analyzed running information and sends the desensitized running data to the data deduplication module for deduplication processing.

Specifically, after receiving the analyzed running information, the data desensitization module identifies the data which may be sensitive in the running information, and replaces the data which may be sensitive by using the characteristic characters, thereby implementing desensitization processing on the running information. The characteristic character may be understood as a preset target character in the above embodiments.

After desensitization processing of the running information is completed, the data desensitization module sends the desensitized running data to the data deduplication module.

Step 2210: the data deduplication module performs deduplication processing on the received operation data and sends the deduplication processed operation data to the cloud database.

Step 2212: the data deduplication module performs deduplication processing on the received operation data and sends the operation data subjected to deduplication processing to the data processing module for processing.

Step 2214: and the data processing module processes the received operation data, determines the operation state data of the dynamic collector and the dynamic collector, and sends the operation state data to the cloud data.

The running state data comprises the dynamic collector, the collection time of the dynamic collector, the collection times and the average collection time in a preset time period, and the success rate of the dynamic collector. The preset time period may be set according to practical applications, and this is not limited in this embodiment of the specification, for example, a day and a week.

Step 240: the dynamic collector acquires the system dynamic information of the client application in a periodic collection mode and sends the system dynamic information to the dynamic collector.

The system dynamic information may be understood as the dynamic operation information in the above embodiments. The system dynamic information can be server disk information, JVM performance information and server memory state information.

The dynamic collector can process the system dynamic information after receiving the system dynamic information sent by the dynamic collector, and sends the processed system dynamic information to the cloud database, and the specific implementation steps comprise:

step 2402: and the dynamic collector processes the received system dynamic information sent by the dynamic collector through the system dynamic information processing module and sends the processed system dynamic information to the cloud database.

Specifically, the dynamic collector formats the system dynamic information through the system dynamic information processing module, converts the format of the system dynamic information into a uniform data format, and sends the processed system dynamic information to the cloud database for storage.

Step 260: and the cloud database sends the operation information and the system dynamic information to the application evaluation platform for evaluation processing.

Specifically, the cloud database sends the operation information and the system dynamic information to an application evaluation platform, and the operation information is evaluated and processed through the application evaluation platform, so that the workload of migration of the heterogeneous database is evaluated.

And performing performance detection processing on the system dynamic information through the application evaluation platform, thereby detecting the performance information of the client application.

In the embodiment of the specification, the running data of the client application is acquired through the dynamic collector, and the running data is subjected to analysis processing, desensitization processing and deduplication processing through the dynamic collector to obtain processed running data; therefore, the real and effective SQL sent by the obtained SQL sentence for the application is clearly determined, and the specific application is known to be initiated, so that the migration efficiency of the heterogeneous database is improved.

Corresponding to the above method embodiment, this specification further provides a data processing system embodiment, and fig. 3 shows a schematic structural diagram of a data processing system provided in an embodiment of this specification. As shown in fig. 3, the system includes a data acquisition device 302 applied to a target server and a data collection device 304 applied to a data processing side different from the target server, wherein,

the data acquisition device 302 is configured to acquire initial operation data of a target application and corresponding data content, determine target operation data based on the data content, and send the target operation data to the data collection device;

the data collection device 304 is configured to parse the received target operation data, determine a characteristic parameter in the target operation data, and process the target operation data based on the characteristic parameter to obtain target data.

Optionally, the data content comprises a data identity;

correspondingly, the data acquisition device 302 is further configured to determine the initial operation data as the target operation data if the data identifier of the initial operation data is consistent with the preset data identifier.

Optionally, the data acquisition device 302 is further configured to determine a data format of the initial operation data based on the data content, and obtain target operation data from the initial operation data based on the data format.

Optionally, the data collection device 304 is further configured to determine identification information corresponding to the characteristic parameter, and process the target operation data based on the characteristic parameter and the identification information corresponding to the characteristic parameter to obtain target data.

Optionally, the data collection apparatus 304 is further configured to:

processing the target operation data based on the characteristic parameters and the identification information to obtain preprocessed data;

and carrying out duplicate removal processing on the preprocessed data to obtain target data.

Optionally, the data collecting apparatus 304 is further configured to:

determining the characteristic parameter corresponding to the identification information as a target characteristic parameter under the condition that the identification information corresponding to the characteristic parameter meets a preset identification condition;

and replacing the target characteristic parameters in the target operation data based on preset target characters to obtain preprocessed data.

Optionally, the data collecting device 304 is further configured to perform format processing on the target operation data, obtain processed target operation data, and store the processed target operation data in an application database.

Optionally, the data collecting device 304 is further configured to obtain attribute information of the target data, and determine status data of the data collecting device and the data collecting device based on the attribute information.

Optionally, the data collection device 304 is further configured to upload the target data and the status data to a target database, so as to perform compatibility detection on the target data in the target database.

Optionally, the data collection apparatus 304 is further configured to:

acquiring dynamic operation information of a target application, and sending the dynamic operation information to the data collection device;

and processing the received dynamic operation information, and sending the processed dynamic operation information to a target database to realize performance detection on the target application in the target database.

The data processing system provided by the specification obtains target operation data based on data content of initial operation data generated in an operation process of a target application in a target server through a data acquisition device applied to the target server, effectively avoids the problem that whether the execution statement acquired from a database cannot be distinguished is an execution statement initiated by the application, can accurately and quickly obtain the execution statement sent by the application, and performs heterogeneous processing on the target operation data through a data acquisition device applied to a data processing end different from the target server after obtaining the target operation data to obtain the target data; thereby reducing the influence on the application and the database and effectively improving the running efficiency of the application and the database

The above is a schematic scheme of a data processing system of the present embodiment. It should be noted that the technical solution of the data processing system and the technical solution of the data processing method belong to the same concept, and details that are not described in detail in the technical solution of the data processing system can be referred to the description of the technical solution of the data processing method.

FIG. 4 illustrates a block diagram of a computing device 400 provided according to an embodiment of the present description. The components of the computing device 400 include, but are not limited to, a memory 410 and a processor 420. Processor 420 is coupled to memory 410 via bus 430 and database 450 is used to store data.

Computing device 400 also includes access device 440, access device 440 enabling computing device 400 to communicate via one or more networks 460. Examples of such networks include the Public Switched Telephone Network (PSTN), a Local Area Network (LAN), a Wide Area Network (WAN), a Personal Area Network (PAN), or a combination of communication networks such as the internet. The access device 440 may include one or more of any type of network interface (e.g., a Network Interface Card (NIC)) whether wired or wireless, such as an IEEE802.11 Wireless Local Area Network (WLAN) wireless interface, a worldwide interoperability for microwave access (Wi-MAX) interface, an ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a bluetooth interface, a Near Field Communication (NFC) interface, and so forth.

In one embodiment of the present description, the above-described components of computing device 400, as well as other components not shown in FIG. 4, may also be connected to each other, such as by a bus. It should be understood that the block diagram of the computing device architecture shown in FIG. 4 is for purposes of example only and is not limiting as to the scope of the present description. Those skilled in the art may add or replace other components as desired.

Computing device 400 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., tablet, personal digital assistant, laptop, notebook, netbook, etc.), mobile phone (e.g., smartphone), wearable computing device (e.g., smartwatch, smartglasses, etc.), or other type of mobile device, or a stationary computing device such as a desktop computer or PC. Computing device 400 may also be a mobile or stationary server.

Wherein the processor 420 is configured to execute computer-executable instructions that, when executed by the processor 420, implement the steps of any of the data processing methods.

The above is an illustrative scheme of a computing device of the present embodiment. It should be noted that the technical solution of the computing device and the technical solution of the data processing method belong to the same concept, and details that are not described in detail in the technical solution of the computing device can be referred to the description of the technical solution of the data processing method.

An embodiment of the present specification also provides a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of any of the data processing methods.

The above is an illustrative scheme of a computer-readable storage medium of the present embodiment. It should be noted that the technical solution of the storage medium belongs to the same concept as the technical solution of the data processing method, and details that are not described in detail in the technical solution of the storage medium can be referred to the description of the technical solution of the data processing method.

An embodiment of the present specification also provides a computer program, wherein when the computer program is executed in a computer, the computer is caused to execute the steps of any of the data processing methods.

The above is an illustrative scheme of a computer program of the present embodiment. It should be noted that the schematic plan of the computer program and the technical plan of the data processing method described above belong to the same concept, and details that are not described in detail in the schematic plan of the computer program can be referred to the description of the technical plan of the data processing method described above.

The foregoing description has been directed to specific embodiments of this disclosure. Other embodiments are within the scope of the following claims. In some cases, the actions or steps recited in the claims may be performed in a different order than in the embodiments and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some embodiments, multitasking and parallel processing may also be possible or may be advantageous.

The computer instructions comprise computer program code which may be in the form of source code, object code, an executable file or some intermediate form, or the like. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, usb disk, removable hard disk, magnetic disk, optical disk, computer Memory, Read-Only Memory (ROM), Random Access Memory (RAM), electrical carrier wave signals, telecommunications signals, software distribution medium, etc. It should be noted that the computer readable medium may contain content that is subject to appropriate increase or decrease as required by legislation and patent practice in jurisdictions, for example, in some jurisdictions, computer readable media does not include electrical carrier signals and telecommunications signals as is required by legislation and patent practice.

It should be noted that, for the sake of simplicity, the foregoing method embodiments are described as a series of acts or combinations, but those skilled in the art should understand that the present disclosure is not limited by the described order of acts, as some steps may be performed in other orders or simultaneously according to the present disclosure. Further, those skilled in the art should also appreciate that the embodiments described in this specification are preferred embodiments and that acts and modules referred to are not necessarily required for this description.

In the above embodiments, the descriptions of the respective embodiments have respective emphasis, and for parts that are not described in detail in a certain embodiment, reference may be made to related descriptions of other embodiments.

The preferred embodiments of the present specification disclosed above are intended only to aid in the description of the specification. Alternative embodiments are not exhaustive and do not limit the invention to the precise embodiments described. Obviously, many modifications and variations are possible in light of the above teaching. The embodiments were chosen and described in order to best explain the principles of the specification and its practical application, to thereby enable others skilled in the art to best understand the specification and its practical application. The specification is limited only by the claims and their full scope and equivalents.

Claims

1. A data processing method is applied to a data processing system, the system comprises a data acquisition device applied to a target server and a data collection device applied to a data processing end different from the target server, wherein,

2. The data processing method of claim 1, the data content comprising a data identification;

3. The data processing method of claim 1, the data acquisition device determining targeted operational data based on the data content, comprising:

4. The data processing method of claim 1, wherein the data collection device processes the target operation data based on the characteristic parameter to obtain target data, and comprises:

5. The data processing method of claim 4, wherein the data collection device processes the target operation data based on the characteristic parameter and the identification information to obtain target data, and comprises:

6. The data processing method of claim 5, wherein the data collection device processes the target operation data based on the characteristic parameter and the identification information to obtain pre-processed data, and comprises:

7. The data processing method of claim 1, after the data collection device determines the characteristic parameter in the target operational data, further comprising:

8. The data processing method of claim 1, wherein the data collection device processes the target operation data based on the characteristic parameter, and after obtaining the target data, the data collection device further comprises:

9. The data processing method of claim 8, further comprising:

10. The data processing method of claim 1, further comprising:

11. A data processing system comprises a data acquisition device applied to a target server and a data collection device applied to a data processing terminal different from the target server,

12. A computing device, comprising:

a memory and a processor;

the memory is for storing computer-executable instructions, and the processor is for executing the computer-executable instructions, which when executed by the processor, implement the steps of the data processing method of any one of claims 1 to 10.

13. A computer-readable storage medium storing computer-executable instructions which, when executed by a processor, implement the steps of the data processing method of any one of claims 1 to 10.

14. A computer program for causing a computer to carry out the steps of the data processing method according to any one of claims 1 to 10 when said computer program is carried out on the computer.