CN107402869B

CN107402869B - Data acquisition method, device and system

Info

Publication number: CN107402869B
Application number: CN201710566404.0A
Authority: CN
Inventors: 王占
Original assignee: Neusoft Corp
Current assignee: Neusoft Corp
Priority date: 2017-07-12
Filing date: 2017-07-12
Publication date: 2021-01-01
Anticipated expiration: 2037-07-12
Also published as: CN107402869A

Abstract

The invention provides a data acquisition method, a device and a system, wherein the method comprises the following steps: acquiring monitoring data of an application program when an access request is acquired; the access request is generated by the application program when the user accesses the application program; stopping the acquisition of the monitoring data after the application program finishes responding to the access request; adding a language tag corresponding to a programming language to the monitoring data according to the programming language adopted by the monitoring data to obtain push data; the language tag is used for indicating the data format of the monitoring data to the data acquisition unit so that the data acquisition unit can convert the data format according to the language tag; pushing the push data to a data acquisition unit; wherein the data collector is independent of the application program. The problem of current because wait for data collection station to gather the control data passively, need occupy local resource and store data, cause to occupy too much application local resource is solved.

Description

Data acquisition method, device and system

Technical Field

The invention relates to the technical field of computers, in particular to a data acquisition method, a data acquisition device and a data acquisition system.

Background

When an application program runs, in order to facilitate monitoring, probes are often added to the application program, so that monitoring data of the application program are acquired through the probes.

However, in the prior art, on one hand, the probe needs to wait for the data acquisition unit to acquire data, so that the probe occupies local resources of the application program to store the data, which is not beneficial to the operation of the application program; on the other hand, because the data formats of the data collected by the probes in different languages are different, the probes also need to perform corresponding data format conversion locally in the application program. Therefore, in the prior art, the probe overhead is large, and excessive local resources of the application program are occupied.

Disclosure of Invention

The present invention is directed to solving, at least to some extent, one of the technical problems in the related art.

Therefore, a first object of the present invention is to provide a data acquisition method, in which a language tag corresponding to a programming language is added to monitoring data according to the programming language used by the monitoring data, and the monitoring data with the language tag added is pushed to a data acquisition unit, so as to actively push the monitoring data to the data acquisition unit, so as to solve the problem that the existing data acquisition unit needs to occupy local resources to store data and occupy too many local resources of an application program because the data acquisition unit is passively waited for acquiring the monitoring data.

A second objective of the present invention is to provide a data acquisition method.

A third object of the invention is to propose a data set system.

A fourth object of the present invention is to provide a data acquisition apparatus.

A fifth object of the present invention is to provide a data acquisition apparatus.

A sixth object of the invention is to propose a computer device.

A seventh object of the invention is to propose a non-transitory computer-readable storage medium.

An eighth object of the invention is to propose a computer program product.

To achieve the above object, an embodiment of a first aspect of the present invention provides a data acquisition method, including: acquiring monitoring data of an application program when an access request is acquired; the access request is generated by the application program when the user accesses the application program; stopping the acquisition of the monitoring data after the application program finishes responding to the access request; adding a language tag corresponding to a programming language to the monitoring data according to the programming language adopted by the monitoring data to obtain push data; the language tag is used for indicating the data format of the monitoring data to the data acquisition unit so that the data acquisition unit can convert the data format according to the language tag; pushing the push data to a data acquisition unit; wherein the data collector is independent of the application program.

According to the data acquisition method provided by the embodiment of the invention, the monitoring data of the application program is acquired when the access request is acquired, the acquisition of the monitoring data is stopped when the application program finishes responding to the access request, the language tag corresponding to the programming language is added to the monitoring data according to the programming language adopted by the acquired monitoring data so as to obtain the push data, and the push data is pushed to the data acquisition unit. Therefore, the monitoring data can be actively pushed to the data acquisition unit, and the problem that the local resources of the application program are excessively occupied due to the fact that the data acquisition unit needs to occupy the local resources to store the data when the data acquisition unit is passively waited for to acquire the monitoring data in the prior art is solved.

In addition, the data acquisition method provided by the embodiment of the invention has the following additional technical characteristics:

in an embodiment of the present invention, after stopping the collection of the monitoring data, the method further includes: adding a corresponding suffix in the index name of the monitoring data according to the data type of the monitoring data; and the suffix is used for indicating the data type required to be adopted when the monitoring data is stored to the data collector.

In an embodiment of the present invention, after stopping the collection of the monitoring data, the method further includes: adding a category mark corresponding to the data category for the monitoring data according to the data category of the monitoring data; and the category mark is used for indicating the processing flow required by the monitoring data to the data acquisition unit.

In an embodiment of the present invention, pushing data to a data collector includes: carrying out protocol processing on the push data by adopting a preset transmission protocol; and pushing the pushed data after the protocol processing to a data acquisition unit by adopting a preset transmission mode.

In order to achieve the above object, a second aspect of the present invention provides a data acquisition method, including: receiving push data pushed by a probe; the push data comprises monitoring data of the application program and a language tag of the monitoring data; wherein the probe is arranged in an application program; inquiring a data format corresponding to the language tag according to the language tag of the monitoring data; and converting the monitoring data into a target format according to the inquired data format.

According to the data acquisition method provided by the embodiment of the invention, the pushed data pushed by the probe is received, the data format corresponding to the language tag is inquired according to the language tag of the monitored data, and the monitored data is converted into the target format according to the inquired data format. Therefore, the monitoring data are converted into the uniform data format according to the language tags, and the technical problem that in the prior art, due to the fact that monitoring data collected by probes in different languages are different in format, corresponding data format conversion needs to be carried out locally on an application program, and probe overhead is large is solved.

in an embodiment of the present invention, the pushing data further includes a category label of the monitoring data, and after receiving the pushing data pushed by the probe, the method further includes: reading a category label of the monitoring data; and processing the monitoring data by adopting a corresponding processing flow according to the class mark.

In one embodiment of the invention, the category labels comprise at least one of index data, transaction data and exception information data; the processing flow corresponding to the index data comprises the following steps: caching the monitoring data; counting the monitoring data cached in a preset time period to obtain a statistical value, wherein the statistical value comprises an average value and/or a variance; the processing flow corresponding to the transaction data comprises the following steps: expanding and/or adding a time stamp to the monitoring data; the processing flow corresponding to the abnormal information data comprises the following steps: the monitoring data is flushed, compressed, and/or time stamped.

In an embodiment of the present invention, after converting the monitoring data into the target format, the method further includes: reading a suffix of an index name of the monitoring data; querying the configuration file according to the suffix to determine the data type required to be adopted when monitoring data storage; and performing data type conversion on the monitoring data according to the determined data type.

In order to achieve the above object, a third embodiment of the present invention provides a data acquisition system, including: a probe for performing the data acquisition method as described in the embodiments of the first aspect; and the data acquisition unit is used for executing the data acquisition method in the embodiment of the second aspect.

According to the data acquisition system provided by the embodiment of the invention, monitoring data in a time period from the time when an access request of an application program is acquired through a probe to the time when the application program finishes responding to the access request are acquired, a language tag corresponding to a programming language is added to the monitoring data according to the programming language adopted by the monitoring data to obtain push data, the push data is pushed to a data acquisition unit, and after the data acquisition unit receives the push data, the data format corresponding to the language tag is inquired according to the language tag of the monitoring data in the push data, and the monitoring data is converted into a target format. Therefore, the monitoring data are converted into the uniform data format according to the language tags, the monitoring data are actively pushed to the data acquisition unit, and the problems that the monitoring data acquired by the probes in different languages are different in format, corresponding data format conversion needs to be carried out locally in an application program, and the data need to be stored by occupying local resources due to the fact that the data acquisition unit is passively waited for acquiring the monitoring data, so that probe overhead is high, and excessive local resources of the application program are occupied are solved.

In addition, the data acquisition system provided by the embodiment of the invention has the following additional technical characteristics:

in one embodiment of the invention, the data acquisition system further comprises: the data memory is used for storing the monitoring data processed by the data acquisition unit; the data analyzer is used for carrying out data analysis on the monitoring data stored in the data storage; and the display is used for displaying the data analysis result obtained by the data analysis of the data analyzer.

To achieve the above object, a fourth aspect of the present invention provides a data acquisition apparatus, including: the acquisition module is used for acquiring monitoring data of the application program when the access request is acquired; the access request is generated by the application program when the user accesses the application program; stopping the acquisition of the monitored data after the application program finishes responding to the access request; the labeling module is used for adding a language label corresponding to a programming language to the monitoring data according to the programming language adopted by the monitoring data to obtain pushing data; the language tag is used for indicating the data format of the monitoring data to the data acquisition unit so that the data acquisition unit can convert the data format according to the language tag; the pushing module is used for pushing the pushing data to the data acquisition unit; wherein the data collector is independent of the application program.

According to the data acquisition device provided by the embodiment of the invention, the monitoring data of the application program is acquired when the access request is acquired, the acquisition of the monitoring data is stopped when the application program finishes responding to the access request, the language tag corresponding to the programming language is added to the monitoring data according to the programming language adopted by the acquired monitoring data so as to obtain the push data, and the push data is pushed to the data acquisition unit. Therefore, the monitoring data can be actively pushed to the data acquisition unit, and the problem that the local resources of the application program are excessively occupied due to the fact that the data acquisition unit needs to occupy the local resources to store the data when the data acquisition unit is passively waited for to acquire the monitoring data in the prior art is solved.

To achieve the above object, a fifth embodiment of the present invention provides a data acquisition device, including: the receiving module is used for receiving the push data pushed by the probe; the push data comprises monitoring data of the application program and a language tag of the monitoring data; wherein the probe is disposed in the application; the query module is used for querying a data format corresponding to the language tag according to the language tag of the monitoring data; and the conversion module is used for converting the monitoring data into a target format according to the queried data format.

According to the data acquisition device provided by the embodiment of the invention, the pushed data pushed by the probe is received, the data format corresponding to the language tag is inquired according to the language tag of the monitored data, and the monitored data is converted into the target format according to the inquired data format. Therefore, the monitoring data are converted into the uniform data format according to the language tags, and the technical problem that in the prior art, due to the fact that monitoring data collected by probes in different languages are different in format, corresponding data format conversion needs to be carried out locally on an application program, and probe overhead is large is solved.

In order to achieve the above object, a sixth aspect of the present invention provides a computer device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, where the processor executes the computer program to implement the data acquisition method according to the first aspect or the data acquisition method according to the second aspect.

To achieve the above object, a seventh embodiment of the present invention proposes a non-transitory computer-readable storage medium, on which a computer program is stored, where the computer program, when executed by a processor, implements the data acquisition method according to the first embodiment or the data acquisition method according to the second embodiment.

To achieve the above object, an eighth aspect of the present invention provides a computer program product, wherein when the instructions in the computer program product are executed by a processor, the data acquisition method according to the embodiment of the first aspect or the data acquisition method according to the embodiment of the second aspect is performed.

Additional aspects and advantages of the invention will be set forth in part in the description which follows and, in part, will be obvious from the description, or may be learned by practice of the invention.

Drawings

The foregoing and/or additional aspects and advantages of the present invention will become apparent and readily appreciated from the following description of the embodiments, taken in conjunction with the accompanying drawings of which:

fig. 1 is a schematic flow chart of a data acquisition method according to an embodiment of the present invention;

fig. 2 is a schematic flow chart illustrating pushing data to a data collector according to an embodiment of the present invention;

FIG. 3 is a schematic flow chart of another data acquisition method according to an embodiment of the present invention;

fig. 4 is a schematic diagram illustrating data types commonly used in a dynamic field matching method according to an embodiment of the present invention;

fig. 5 is a schematic flow chart of a specific data acquisition method according to an embodiment of the present invention;

fig. 6 is a schematic flow chart of another data acquisition method according to an embodiment of the present invention;

fig. 7 is a schematic flow chart of a data collector receiving push data according to an embodiment of the present invention;

fig. 8 is a schematic flow chart illustrating a further data acquisition method according to an embodiment of the present invention;

FIG. 9 is a diagram illustrating a process of data type conversion using dynamic field matching rules according to an embodiment of the present invention;

fig. 10 is a schematic diagram of an application scenario for converting data types by using dynamic matching rules according to an embodiment of the present invention;

FIG. 11 is a diagram illustrating a comparison of a data type conversion method according to an embodiment of the present invention;

fig. 12 is a schematic flow chart illustrating a specific data acquisition method according to an embodiment of the present invention;

fig. 13 is a schematic structural diagram of a data acquisition system according to an embodiment of the present invention;

fig. 14 is a schematic structural diagram of another data acquisition system according to an embodiment of the present invention;

fig. 15 is a functional schematic diagram of a data acquisition system according to an embodiment of the present invention;

FIG. 16 is a schematic structural diagram of a data acquisition device according to an embodiment of the present invention;

fig. 17 is a schematic structural diagram of another data acquisition device according to an embodiment of the present invention.

Detailed Description

Reference will now be made in detail to embodiments of the present invention, examples of which are illustrated in the accompanying drawings, wherein like or similar reference numerals refer to the same or similar elements or elements having the same or similar function throughout. The embodiments described below with reference to the drawings are illustrative and intended to be illustrative of the invention and are not to be construed as limiting the invention.

The data acquisition method, apparatus and system of the embodiments of the present invention are described below with reference to the accompanying drawings.

However, in the prior art, on one hand, the probe needs to wait for the data acquisition unit to acquire data, so that the probe occupies local resources of the application program to store the data, which is not beneficial to the operation of the application program; on the other hand, because the data formats of the monitoring data collected by the probes in different languages are different, the probes also need to perform corresponding data format conversion locally in the application program. Therefore, in the prior art, the probe overhead is large, and excessive local resources of the application program are occupied.

In order to solve the problem, an embodiment of the present invention provides a data acquisition method, which adds a language tag corresponding to a programming language to monitoring data according to the programming language adopted by the monitoring data, and pushes the monitoring data with the language tag added to a data acquisition unit, so as to actively push the monitoring data to the data acquisition unit, so as to solve the problem that the existing data acquisition unit needs to occupy local resources to store data and occupy too many local resources of an application program because the data acquisition unit is passively waited for acquiring the monitoring data.

Fig. 1 is a schematic flow chart of a data acquisition method according to an embodiment of the present invention. The data acquisition method proposed by the embodiment of the present invention is described below from the probe side with reference to fig. 1.

As shown in fig. 1, the data acquisition method includes the following steps:

s101, when the access request is acquired, monitoring data of the application program are acquired.

Wherein the access request is generated by the application when the user accesses the application. For example, when the user opens the payment instrument, the payment instrument will generate an access request for the user to access the payment instrument, and respond according to the access request.

In an embodiment of the present invention, a probe may be set in an application in advance to collect monitoring data of the application through the probe. Wherein, different types of probes, such as Java probe, PHP probe,. Net probe, etc., can be set according to different programming languages.

Specifically, when a user opens an application, the application generates a corresponding access request. When the probe acquires the access request, the probe starts to collect monitoring data of the application program, such as index data for measuring the performance of the application program, transaction data and the like.

And S102, stopping the acquisition of the monitoring data after the application program finishes responding to the access request.

In an embodiment of the invention, the probe collects monitoring data from the time when the access request is acquired to the time when the application finishes responding to the access request.

S103, adding a language tag corresponding to the programming language for the monitoring data according to the programming language adopted by the monitoring data to obtain the push data.

The language tag is used for indicating the data format of the monitoring data to the data acquisition unit so that the data acquisition unit can convert the data format according to the language tag.

Because different programming languages have different syntax rules, the data formats of the monitoring data adopting different programming languages are different. In order to unify data formats of monitoring data and facilitate storage in the same data format, embodiments of the present invention provide that a language tag is added to monitoring data in different programming languages.

Specifically, after the collection of the monitoring data is stopped, a language tag corresponding to a programming language is added to the monitoring data according to the programming language adopted by the collected monitoring data, so that the pushed data including the monitoring data and the language tag of the monitoring data is obtained.

For example, if the index data is Java language and the transaction data is C language in the collected monitoring data, language tags corresponding to the Java language and the C language are added to the index data and the transaction data, respectively, so that the data collector can search the programming language used by the monitoring data according to the language tags, and convert the data format corresponding to the programming language into a uniform data format.

And S104, pushing the push data to the data acquisition unit.

Wherein the data collector is independent of the application program. That is, unlike the probe set in the application, the data collector's functional implementation is independent of the application.

After the language tags corresponding to the programming languages are added to the monitoring data adopting different programming languages, the probe pushes the pushed data to the data acquisition unit through the data sending interface.

Specifically, a transmission protocol and a preset transmission mode used by the push data are preset. And when the probe generates the push data, carrying out protocol processing on the push data by adopting a preset transmission protocol. And then, pushing the pushed data after the protocol processing to a data acquisition unit through a data sending interface by adopting a preset transmission mode.

As an example, the flow of actively pushing push data is shown in fig. 2. Firstly, establishing a data transmission mode, such as transmission by using blocking I/O; then, a data transmission protocol is established, and the data is compressed by using a binary coding format so as to improve the data transmission efficiency; and finally, a data sending end is established based on the transmission mode and the transmission protocol, and compressed push data is sent to the data acquisition unit through the sending end.

In the embodiment of the invention, the probe actively pushes the pushed data to the data collector, and compared with the prior art that the probe needs to wait for the data collector to collect the data passively, the probe occupies local resources of an application program to store the data, and does not need to occupy excessive local resources of the application program.

Alternatively, as shown in fig. 3, after stopping the collection of the monitoring data, the data collection method may further include step S105.

And S105, adding a corresponding suffix in the index name of the monitoring data according to the data type of the monitoring data.

The index name refers to the name of the performance index of the application program, and for example, the index name of the http state code is http _ code. And the suffix is used for indicating the data type required to be adopted when the monitoring data is stored to the data collector.

Due to differences in data types between different programming languages, problems may arise in data transmission between probes and data collectors having different programming language categories. For example, Javascript (JS for short) and PHP belong to a weak type language (data types can be ignored), and the data types do not need to be specified when defining the index names; the JAVA and C languages belong to a strongly typed language (all variables must be defined and then used), and the data type needs to be specified when defining the index name.

In addition, when the probe pushes data to the data collector, due to the difference between the programming language used by the monitoring data collected by the probe and the programming language used by the data collector, the data type received by the data collector is not the desired data type, and at this time, data type conversion is required. For example, the JS probe acquires monitoring data in JS language and the data acquisition device in SCALA language, and data type conversion is required.

In order to unify conversion standards of data types of different programming languages, an embodiment of the present invention provides that, when an index name of monitoring data is defined, a data collector converts a data type according to the suffix of the index name by adding the suffix to the index name, so that the monitoring data is finally stored in the converted data type.

In one possible implementation form of the invention, a suffix can be added to the index name by using the dynamic field matching rule, and the data type of the monitoring data storage is directly specified by the suffix. As shown in fig. 4, a suffix _ is may be used to indicate that the data type of the monitored data corresponds to an int array; with suffix _ dt, the data type indicating the monitoring data corresponds to date and so on.

In the embodiment of the invention, the suffix is added in the index name of the monitoring data according to the data type of the monitoring data, and the final storage type of the monitoring data is designated, so that the data type finally stored by the monitoring data is designated when the index name is defined, the conversion standard of the data type is unified, and the data problem caused by the difference of the data types of different programming languages can be effectively avoided.

Alternatively, as shown in fig. 3, after stopping the collection of the monitoring data, the data collection method may further include step S106.

And S106, adding a category mark corresponding to the data category for the monitoring data according to the data category of the monitoring data.

The category mark is used for indicating a processing flow required by the monitoring data to the data collector.

The data types of the collected monitoring data include but are not limited to index data, transaction data, abnormal information data and the like during the running process of the application program. And, for the monitoring data of different data categories, the required processing flow is different.

Therefore, in order to distinguish the monitoring data of different data types, in the embodiment of the present invention, a type label corresponding to the data type is added to the monitoring data according to the data type of the monitoring data, so that the data collector performs a corresponding processing flow on the monitoring data according to the type label.

In addition, the execution sequence of steps S103, 105, and 106 is not limited in the embodiment of the present invention.

The data acquisition method provided by the embodiment of the invention is described in detail by a specific embodiment.

As shown in fig. 5, the data acquisition method includes:

s501, setting a sampling frequency.

When the sampling frequency is set, full sampling or partial sampling can be set. For example, during the running process of an application program, there are 10 monitoring data, and full sampling can be set, that is, 10 monitoring data are acquired by a probe.

In practical application, it is generally not necessary to acquire all monitoring data, and the variation trend of the application program performance can be obtained only by acquiring a part of the monitoring data, and at this time, partial sampling can be set. If there are 10 monitoring data, the probe can be set to acquire 2 monitoring data.

By setting full sampling or partial sampling, different concurrent requirements are met, and the influence of the probe on the application program can be reduced.

And S502, registering a probe.

In the embodiment of the invention, probes of multiple programming language types, such as a JAVA probe, a PHP probe, a NET probe and the like, can be added and registered in an application program to be monitored, so as to support collection of monitoring data of multiple programming languages.

S503, an access request is generated.

When a user opens an application, the application generates an access request. For example, when a pay is turned on for a transfer, the application generates a transfer access request.

And S504, collecting monitoring data.

When an access request generated by the application program is acquired, the probe acquires monitoring data in the running process of the application program according to the set sampling frequency until the application program finishes responding to the access request. The collected monitoring data may be the time elapsed from the acquisition of the access request to the start of the response to the access request, and the like.

In order to unify conversion standards of data types of different programming languages, the embodiment of the invention provides that when an index name is defined, a suffix is added in the index name through a dynamic field matching rule, so that a data acquisition unit performs data type conversion according to the suffix of the index name, and monitoring data are finally stored in a converted data type.

And S505, adding a data category mark.

In order to distinguish the monitoring data of different data types, a type mark corresponding to the data type is added to the monitoring data according to the data type of the monitoring data, so that the data acquisition unit carries out corresponding processing flow on the monitoring data according to the type mark.

S506, adding a programming language label.

After adding the data category mark to the monitoring data, adding a language tag corresponding to a programming language to the monitoring data according to the programming language adopted by the collected monitoring data, thereby obtaining pushed data comprising the monitoring data and the language tag of the monitoring data.

For example, if the index data is Java language and the transaction data is C language in the collected monitoring data, language tags corresponding to the Java language and the C language are added to the index data and the transaction data, respectively, so that the data collector can obtain the programming language used by the monitoring data according to the language tags and convert the data format corresponding to the programming language into a uniform data format.

And S507, pushing data.

After a language tag corresponding to a programming language is added to the monitoring data, protocol processing is performed on the pushed data through a preset transmission protocol so as to meet the requirements of the transmission protocol. And then, pushing the pushed data processed by the protocol to a data acquisition unit by adopting a preset transmission protocol.

After pushing the pushed data to the data collector, the method may continue to wait for a new access request, and after acquiring the new access request, perform the operations of steps S504-S507.

For clarity of the above embodiment, the present embodiment provides a data acquisition method. Fig. 6 is a schematic flow chart of another data acquisition method according to an embodiment of the present invention.

The data acquisition method proposed by the embodiment of the present invention is described below from the data acquisition side with reference to fig. 6.

As shown in fig. 6, the data acquisition method may include the steps of:

s601, receiving push data pushed by a probe; the push data includes monitoring data of the application program and language tags of the monitoring data.

The probe is arranged in an application program, and probes of multiple programming languages can be arranged to collect monitoring data of multiple programming prologs.

As an example of the data collector receiving the pushed data pushed by the probe, as shown in fig. 7. Firstly, a data acquisition unit creates a data processing interface; then, creating a data processing object based on the data processing interface; then, a data transmission mode and a data transmission protocol are established, and the data transmission mode and the data transmission protocol are consistent with the data probe end; and finally, establishing and starting a data receiving service based on the data processing object, the data transmission mode and the data transmission protocol, wherein in the example, a multithreading standard blocking type I/O mode is selected to be used for receiving the push data.

S602, according to the language tag of the monitoring data, inquiring a data format corresponding to the language tag.

And after receiving the pushed data, the data acquisition unit acquires the language tag corresponding to the monitoring data from the pushed data so as to query the data format corresponding to the language tag according to the language tag of the monitoring data.

For example, if the push data received by the data collector includes 3 pieces of monitoring data, where the language tags corresponding to 2 pieces of monitoring data are JAVA language tags, the data format corresponding to the JAVA language is queried according to the tags.

And S603, converting the monitoring data into a target format according to the inquired data format.

The data acquisition unit converts the monitoring data into a predefined target format according to the data format inquired, so that the monitoring data of different programming languages are converted into a uniform target format.

Further, after the monitoring data is converted into a uniform target format, as shown in fig. 8, the data acquisition method may further include:

s604, reading a suffix of the index name of the monitoring data.

In order to convert the data type of the monitoring data into a defined data type, a suffix of an index name of the monitoring data needs to be read.

And S605, inquiring the configuration file according to the suffix so as to determine the data type required to be adopted when the data storage is monitored.

In the embodiment of the invention, the regular expression can be used for defining the dynamic field matching model in advance to generate the configuration file. After reading the suffix of the index name of the monitoring data, inquiring the configuration file, and matching the index suffix according to the regular matching rule to determine the data type required to be adopted when the monitoring data is stored.

And S606, performing data type conversion on the monitoring data according to the determined data type.

Specifically, a process of data type conversion is performed using a dynamic field matching rule, as shown in fig. 9. First, a configuration file is generated by defining a dynamic field matching model using a regular expression. After receiving the pushed data, reading the configuration file, and searching a suffix matched with the index name suffix of the monitoring data in the configuration file by using a regular expression to determine the data type to which the monitoring data needs to be converted, which is a common data conversion type, as shown in fig. 4, if the index name suffix of the monitoring data is ". times.. dt", the corresponding data type is date, that is, the data type of the monitoring data needs to be converted into date type.

The following describes a scenario of data type conversion using a dynamic field matching rule by referring to fig. 10.

When probes of different programming languages transmit data, the data types of the same index data can be different. As shown in fig. 10, the HTTP status code is defined in JS, and may be assigned to be integer type when being assigned, and the type of the HTTP status code may be directly defined in JAVA, and may be a character string type when data type conversion is required.

In fig. 10, two probes are respectively a JS probe and a JAVA probe, and the data acquisition unit adopts the SCALA programming language. If the http state code is stored in a character string mode, the original data type of the http state code with the programming language JS is an integer type, and when the index name is defined, a suffix _ s is added in the index name by using a dynamic field matching rule, so that the index name is defined as http _ code _ s.

Similarly, for the http status code with the programming language of JAVA, since JAVA is a strong language, when defining the index name, an _ s may be added to the suffix of the index name to specify the data type thereof. After the data acquisition unit receives the monitoring data, the regular expression matching rule is utilized to find out the dynamic field matching ". about.s", and the corresponding data type is a String type. Then, the data collector converts the two types of monitoring data into String type. Finally, the http status code is stored in String type.

For the data type difference problem, a dictionary matching method is usually adopted for conversion, as shown in fig. 11, a file including all index names and corresponding data types is defined, and when the data is received and the type conversion is needed, the whole file is traversed to search the index names and the corresponding data types. However, this data type conversion method is inefficient, and when the index name is added, the matching correspondence relationship of the dictionary file also needs to be updated.

As shown in fig. 11, in the dynamic field matching method provided in the embodiment of the present invention, only a probe is required to add a specified suffix to an index name of monitoring data (if data type conversion is not required, no suffix is required to be added), and after a data collector receives the monitoring data, a matched field and a corresponding data type are searched in a configuration file in which a dynamic field is defined by a regular expression, and the monitoring data is converted into the specified data type, so that the type finally output by the data collector is the specified data type.

In addition, the method not only improves the conversion efficiency of the data types, but also can effectively solve the problem caused by the difference of the data types when monitoring data of different programming languages are received without frequently updating the configuration files.

In summary, the monitoring data can be converted into the data type used in storage through the suffix of the monitoring data index name and the dynamic field matching rule.

Furthermore, the pushed data received by the data collector may further include a category label of the monitoring data, and different processing flows are performed on the monitoring data according to the category label. The data acquisition method proposed by the embodiment of the present invention is described in detail below with reference to fig. 12.

First, the data collector starts a service for receiving monitoring data to receive monitoring data of different programming language types. Then, the type of the monitoring data is judged through a calling interface.

If the monitoring data are index data, caching the monitoring data, counting the monitoring data within a preset time period to obtain statistical values such as an average value, a variance, a standard deviation and the like, converting the data format of the monitoring data into a uniform data format according to a language tag in the monitoring data, converting the monitoring data into an appointed data type by using a dynamic field matching rule, and finally storing the data through a data sending interface.

And if the monitoring data is transaction data, expanding the monitoring data, such as expanding information of geographic positions according to the IP address, adding a timestamp, converting the data format of the monitoring data into a uniform data format according to a language tag in the monitoring data, converting the monitoring data into a specified data type by using a dynamic field matching rule, and finally storing the data through a data sending interface.

If the monitoring data is abnormal information data, the monitoring data is cleaned, unnecessary information is removed, then compression is carried out, a timestamp is added, then the data format of the monitoring data is converted into a uniform target format according to a language tag in the monitoring data, then the monitoring data is converted into a specified data type by using a dynamic field matching rule, and finally the monitoring data is stored through a data sending interface. The abnormal information data is cleaned and compressed, so that the size of the abnormal information data can be reduced, and the storage space is saved.

And if the monitored data is not the three data, the monitored data is regarded as abnormal data. The abnormal data processing flow is to count and store the abnormal data so as to record the state of the probe as a reference to achieve the effect of managing the probe, generate warning information according to the abnormal data and send the warning information through a data expansion interface. In addition, the data expansion interface can also be used for performing small-segment capture analysis, abnormal data statistics, probe state record analysis and the like on the data.

In order to implement the above embodiments, the present invention further provides a data acquisition system.

As shown in fig. 13, the data acquisition system includes: probes 1310, data collector 1320.

A probe 1310 for performing the probe-side data acquisition method as described in the above embodiments.

The data collector 1320 is configured to perform the data collection method of the data collector side according to the embodiment.

Further, as shown in fig. 14, the data acquisition system further includes: data store 1330, data analyzer 1340, presenter 1350.

The data storage 1330 is configured to store the monitoring data processed by the data collector.

The data analyzer 1340 is used to perform data analysis on the monitoring data stored in the data storage.

The display 1350 is configured to display a data analysis result obtained by analyzing data by the data analyzer.

For clarity, the data acquisition system according to the embodiment of the present invention is described in detail with reference to fig. 15.

The probe in the data acquisition system shown in fig. 15 can acquire monitoring data in four languages, i.e., JAVA language, PHP language, JS language, and NET language, and push the acquired monitoring data to the data acquisition unit. The data acquisition unit performs data acquisition service through the data receiving interface and receives monitoring data of different programming language types. The data acquisition processes the monitoring data, such as data statistics or expansion, data format unification, data type conversion, and the like, which is described in the above embodiments and is not described herein again.

And after the data acquisition unit finishes processing the monitoring data, the processed monitoring data is sent to the data storage unit for storage through the data sending interface. And the data memory stores the processed monitoring data after receiving the processed monitoring data. And then, the data analyzer performs data analysis on the monitoring data stored in the data storage and sends the analysis result to the displayer. Finally, the display displays the data analysis result, for example, the analysis result is displayed through a UI interface, so that a user can conveniently know the monitoring result of the application program.

Therefore, it can be seen that, unlike the existing data acquisition system in which probes in different programming languages correspond to one data acquisition device, in the data acquisition system provided in the embodiment of the present invention, probes in different programming languages correspond to one data acquisition device, and the data acquisition device expands some other functions, such as data expansion and dynamic field matching, and opens an expanded data interface, so as to implement functions of data capture, abnormal data statistics, probe state analysis, and the like.

According to the data acquisition system provided by the embodiment of the invention, monitoring data in a time period from the time when an access request of an application program is acquired through a probe to the time when the application program finishes responding to the access request are acquired, a language tag corresponding to a programming language is added to the monitoring data according to the programming language adopted by the monitoring data to obtain push data, the push data is pushed to a data acquisition unit, and after the data acquisition unit receives the push data, the data format corresponding to the language tag is inquired according to the language tag of the monitoring data in the push data, and the monitoring data is converted into a target format. Therefore, the monitoring data are converted into the uniform data format according to the language tags, the monitoring data are actively pushed to the data acquisition unit, and the problems that the monitoring data acquired by the probes in different languages are different in format, corresponding data format conversion needs to be carried out locally in an application program, and the data need to be stored by occupying local resources due to the fact that the data acquisition unit is passively waited for acquiring the monitoring data, so that probe overhead is large, and too many local resources are occupied are solved.

In order to implement the above embodiments, the present invention further provides a data acquisition device.

Fig. 16 is a schematic structural diagram of a data acquisition device according to an embodiment of the present invention.

As shown in fig. 16, the apparatus includes: a collection module 1610, a labeling module 1620, and a pushing module 1630.

The acquiring module 1610 is configured to acquire monitoring data of an application program when an access request is acquired; the access request is generated by the application program when the user accesses the application program; and stopping the collection of the monitoring data after the application program finishes responding to the access request.

The labeling module 1620 is configured to add a language label corresponding to a programming language to the monitoring data according to the programming language used by the monitoring data to obtain push data; the language tag is used for indicating the data format of the monitoring data to the data acquisition unit so that the data acquisition unit can convert the data format according to the language tag.

The pushing module 1630 is configured to push the pushed data to the data collector; wherein the data collector is independent of the application program.

It should be noted that the foregoing explanation of the embodiment of the probe-side data acquisition method is also applicable to the data acquisition apparatus of this embodiment, and is not repeated herein.

In order to realize the embodiment, the invention also provides another data acquisition device.

As shown in fig. 17, the data acquisition apparatus includes: a receiving module 1710, a query module 1720, and a translation module 1730.

The receiving module 1710 is configured to receive push data pushed by the probe; the push data comprises monitoring data of the application program and a language tag of the monitoring data; wherein the probe is provided in the application.

The query module 1720 is configured to query a data format corresponding to a language tag according to the language tag of the monitoring data.

The conversion module 1730 is used for converting the monitoring data into the target format according to the data format queried.

It should be noted that the explanation of the data acquisition method embodiment at the data acquisition unit side is also applicable to the data acquisition device of the embodiment, and is not repeated herein.

In order to implement the foregoing embodiments, an embodiment of the present invention provides a computer device, which includes a memory, a processor, and a computer program that is stored in the memory and is executable on the processor, where when the processor executes the computer program, a data acquisition method is implemented, where the method includes: acquiring monitoring data of an application program when an access request is acquired; the access request is generated by the application program when the user accesses the application program; stopping the acquisition of the monitoring data after the application program finishes responding to the access request; adding a language tag corresponding to a programming language to the monitoring data according to the programming language adopted by the monitoring data to obtain push data; the language tag is used for indicating the data format of the monitoring data to the data acquisition unit so that the data acquisition unit can convert the data format according to the language tag; pushing the push data to a data acquisition unit; wherein the data collector is independent of the application program.

In order to implement the above embodiments, the present invention also proposes a non-transitory computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements a data acquisition method, the method comprising: acquiring monitoring data of an application program when an access request is acquired; the access request is generated by the application program when the user accesses the application program; stopping the acquisition of the monitoring data after the application program finishes responding to the access request; adding a language tag corresponding to a programming language to the monitoring data according to the programming language adopted by the monitoring data to obtain push data; the language tag is used for indicating the data format of the monitoring data to the data acquisition unit so that the data acquisition unit can convert the data format according to the language tag; pushing the push data to a data acquisition unit; wherein the data collector is independent of the application program.

In order to implement the foregoing embodiments, the present invention further provides a computer program product, which when executed by an instruction processor in the computer program product performs a data acquisition method, where the method includes: acquiring monitoring data of an application program when an access request is acquired; the access request is generated by the application program when the user accesses the application program; stopping the acquisition of the monitoring data after the application program finishes responding to the access request; adding a language tag corresponding to a programming language to the monitoring data according to the programming language adopted by the monitoring data to obtain push data; the language tag is used for indicating the data format of the monitoring data to the data acquisition unit so that the data acquisition unit can convert the data format according to the language tag; pushing the push data to a data acquisition unit; wherein the data collector is independent of the application program.

In order to implement the foregoing embodiments, an embodiment of the present invention provides a computer device, which includes a memory, a processor, and a computer program that is stored in the memory and is executable on the processor, where when the processor executes the computer program, a data acquisition method is implemented, where the method includes: receiving push data pushed by a probe; the push data comprises monitoring data of the application program and a language tag of the monitoring data; wherein the probe is arranged in an application program; inquiring a data format corresponding to the language tag according to the language tag of the monitoring data; and converting the monitoring data into a target format according to the inquired data format.

In order to implement the above embodiments, the present invention also proposes a non-transitory computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements a data acquisition method, the method comprising: receiving push data pushed by a probe; the push data comprises monitoring data of the application program and a language tag of the monitoring data; wherein the probe is arranged in an application program; inquiring a data format corresponding to the language tag according to the language tag of the monitoring data; and converting the monitoring data into a target format according to the inquired data format.

In order to implement the foregoing embodiments, the present invention further provides a computer program product, which when executed by an instruction processor in the computer program product performs a data acquisition method, where the method includes: receiving push data pushed by a probe; the push data comprises monitoring data of the application program and a language tag of the monitoring data; wherein the probe is arranged in an application program; inquiring a data format corresponding to the language tag according to the language tag of the monitoring data; and converting the monitoring data into a target format according to the inquired data format.

In the description herein, references to the description of the term "one embodiment," "some embodiments," "an example," "a specific example," or "some examples," etc., mean that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the invention. In this specification, the schematic representations of the terms used above are not necessarily intended to refer to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. Furthermore, various embodiments or examples and features of different embodiments or examples described in this specification can be combined and combined by one skilled in the art without contradiction.

Furthermore, the terms "first", "second" and "first" are used for descriptive purposes only and are not to be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one such feature. In the description of the present invention, "a plurality" means at least two, e.g., two, three, etc., unless specifically limited otherwise.

Any process or method descriptions in flow charts or otherwise described herein may be understood as representing modules, segments, or portions of code which include one or more executable instructions for implementing steps of a custom logic function or process, and alternate implementations are included within the scope of the preferred embodiment of the present invention in which functions may be executed out of order from that shown or discussed, including substantially concurrently or in reverse order, depending on the functionality involved, as would be understood by those reasonably skilled in the art of the present invention.

The logic and/or steps represented in the flowcharts or otherwise described herein, e.g., an ordered listing of executable instructions that can be considered to implement logical functions, can be embodied in any computer-readable medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, processor-containing system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions. For the purposes of this description, a "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of the computer-readable medium would include the following: an electrical connection (electronic device) having one or more wires, a portable computer diskette (magnetic device), a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). Additionally, the computer-readable medium could even be paper or another suitable medium upon which the program is printed, as the program can be electronically captured, via for instance optical scanning of the paper or other medium, then compiled, interpreted or otherwise processed in a suitable manner if necessary, and then stored in a computer memory.

It should be understood that portions of the present invention may be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, the various steps or methods may be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. If implemented in hardware, as in another embodiment, any one or combination of the following techniques, which are known in the art, may be used: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application specific integrated circuit having an appropriate combinational logic gate circuit, a Programmable Gate Array (PGA), a Field Programmable Gate Array (FPGA), or the like.

It will be understood by those skilled in the art that all or part of the steps carried by the method for implementing the above embodiments may be implemented by hardware related to instructions of a program, which may be stored in a computer readable storage medium, and when the program is executed, the program includes one or a combination of the steps of the method embodiments.

In addition, functional units in the embodiments of the present invention may be integrated into one processing module, or each unit may exist alone physically, or two or more units are integrated into one module. The integrated module can be realized in a hardware mode, and can also be realized in a software functional module mode. The integrated module, if implemented in the form of a software functional module and sold or used as a stand-alone product, may also be stored in a computer readable storage medium.

The storage medium mentioned above may be a read-only memory, a magnetic or optical disk, etc. Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention, and that variations, modifications, substitutions and alterations can be made to the above embodiments by those of ordinary skill in the art within the scope of the present invention.

Claims

1. A data acquisition method is characterized by comprising the following steps:

acquiring monitoring data of an application program when an access request is acquired; the access request is generated by an application program when a user accesses the application program;

stopping the acquisition of the monitoring data after the application program finishes responding to the access request, and adding a corresponding suffix in the index name of the monitoring data according to the data type of the monitoring data; the suffix is used for indicating the data type required to be adopted when the monitoring data is stored to a data acquisition unit;

adding a language tag corresponding to a programming language to the monitoring data according to the programming language adopted by the monitoring data to obtain pushing data; the language tag is used for indicating the data format of the monitoring data to the data acquisition unit so that the data acquisition unit can perform data format conversion according to the language tag;

pushing the pushed data to the data acquisition device; wherein the data collector is independent of the application program.

2. The data collection method of claim 1, further comprising, after stopping the collection of the monitoring data:

adding a category label corresponding to the data category to the monitoring data according to the data category of the monitoring data; the category mark is used for indicating the processing flow required by the monitoring data to the data collector.

3. The data acquisition method according to claim 1 or 2, wherein the pushing the pushed data to the data acquisition device comprises:

carrying out protocol processing on the push data by adopting a preset transmission protocol;

and pushing the pushed data after the protocol processing to the data acquisition unit by adopting a preset transmission mode.

4. A data acquisition method is characterized by comprising the following steps:

receiving push data pushed by a probe; the push data comprises monitoring data of an application program and a language tag of the monitoring data; wherein the probe is disposed in the application;

inquiring a data format corresponding to the language tag according to the language tag of the monitoring data;

converting the monitoring data into a target format according to the inquired data format;

reading a suffix of an index name of the monitoring data;

inquiring a configuration file according to the suffix to determine the data type required to be adopted when the monitoring data is stored;

and performing data type conversion on the monitoring data according to the determined data type.

5. The data collection method according to claim 4, wherein the pushed data further includes a category label of the monitoring data, and after receiving the pushed data pushed by the probe, the method further includes:

reading a category label of the monitoring data;

and processing the monitoring data by adopting a corresponding processing flow according to the category mark.

6. The data collection method of claim 5, wherein the category labels comprise at least one of indicator data, transaction data, and anomaly information data;

the processing flow corresponding to the index data comprises the following steps: caching the monitoring data; counting the monitoring data cached in a preset time period to obtain a statistical value, wherein the statistical value comprises an average value and/or a variance;

the processing flow corresponding to the transaction data comprises the following steps: expanding and/or time stamping the monitoring data;

the processing flow corresponding to the abnormal information data comprises the following steps: and cleaning, compressing and/or time stamping the monitoring data.

7. A data acquisition system, characterized in that the system comprises:

a probe for performing the data acquisition method of any one of claims 1-3;

data collector for performing the data collection method of any one of claims 4-6.

8. A data acquisition device, comprising:

the acquisition module is used for acquiring monitoring data of the application program when the access request is acquired; the access request is generated by an application program when a user accesses the application program; stopping the acquisition of the monitoring data after the application program finishes responding to the access request;

the labeling module is used for adding a language label corresponding to a programming language to the monitoring data according to the programming language adopted by the monitoring data to obtain pushing data; the language tag is used for indicating the data format of the monitoring data to a data acquisition unit so that the data acquisition unit can perform data format conversion according to the language tag;

the pushing module is used for pushing the pushed data to the data acquisition unit; wherein the data collector is independent of the application program.

9. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor, when executing the program, implements a data acquisition method as claimed in any one of claims 1 to 3 or a data acquisition method as claimed in any one of claims 4 to 6.

10. A non-transitory computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, implements a data acquisition method according to any one of claims 1 to 3, or a data acquisition method according to any one of claims 4 to 6.

11. A computer program product in which instructions, when executed by a processor, perform a data acquisition method as claimed in any one of claims 1 to 3, or a data acquisition method as claimed in any one of claims 4 to 6.