CN113032221A

CN113032221A - Data acquisition and transmission method and device, computer equipment and storage medium

Info

Publication number: CN113032221A
Application number: CN202110338950.5A
Authority: CN
Inventors: 刘小龙
Original assignee: Shenzhen Hongtu Chuangcheng Technology Co ltd
Current assignee: Shenzhen Hongtu Chuangcheng Technology Co ltd
Priority date: 2021-03-30
Filing date: 2021-03-30
Publication date: 2021-06-25

Abstract

The invention relates to a data acquisition and transmission method, a device, computer equipment and a storage medium, wherein the method comprises the steps of acquiring an acquisition strategy from a management terminal; setting an actual acquisition strategy by adopting a switch mode according to the acquisition strategy; intercepting the access data by adopting a byte enhancement technology to obtain intercepted data; acquiring user behavior data, application transmission data and application asset data of the intercepted data by combining an actual acquisition strategy with a linkage mechanism to obtain target data; caching target data; and carrying out guaranteed transmission on the target data. The invention can realize the controllable collection of user behavior data, transmission data and application asset data, and has the advantages of more comprehensive, simple and flexible collection, high efficiency and low maintenance cost; the collected target data is cached firstly and then is subjected to guaranteed transmission, a high-availability guarantee mechanism is provided during transmission, and performance degradation of the application client is avoided.

Description

Data acquisition and transmission method and device, computer equipment and storage medium

Technical Field

The present invention relates to computers, and more particularly to a method and apparatus for data acquisition and transmission, a computer device, and a storage medium.

Background

When network security and data security management and control are performed, big data real-time analysis, security analysis, traceability analysis, evidence obtaining and the like need to be performed on user behavior data, application transmission data and application asset data, so that scenes acquired by the user behavior data, the application transmission data and the application asset data are gradually increased, and the demand is increased.

At present, the acquisition of user behavior data, application transmission data and application asset data comprises three modes, one mode is to use a preset application program embedded point or a secondary development fixed point acquisition scheme to acquire the user behavior data, the application transmission data and the application asset data, but the mode needs secondary development aiming at each appointed acquisition point, the development and maintenance cost is extremely high, and the invasion exists on the original service because each acquisition needs secondary development, code modification and operation; the switch for data acquisition needs to be maintained on codes, and interactive dynamic configuration and adjustment cannot be performed on data acquisition objects, such as only acquiring user behavior data of a designated user, a designated interface and a designated application service; the second way is to collect application transmission data and application asset data by analyzing network flow, which can not analyze the flow and data content of some private protocols, the network flow analysis can not analyze the flow and data analysis because private protocol fields and standards are not disclosed, and can not analyze the application interface and reduction transmission data of the application system, and can not analyze the encrypted flow, because the deployment mode is limited to the range of the collectable application transmission data, only can partially collect the data accessed by users, and can not collect the data call among application systems, the complexity of the network architecture can be improved, the difficulty of network operation and maintenance can be improved, the efficiency of fault elimination can be reduced, the reliability and usability of the whole network can be reduced, and the objects of data collection can not be managed and controlled by an interactive data collection strategy, such as by frequency of acquisition, data content, etc.; the other way is to collect application transmission data by erecting an application security gateway, which cannot collect applications not included in the application gateway range, can only collect user access data included in the application gateway range, cannot collect data transmission between applications, needs to adjust the existing application system architecture, has a great influence, and cannot manage and control data collection objects through an interactive data collection strategy, such as collection frequency, data content and the like; in the three manners, when data acquisition and processing and data transmission are performed, a large amount of data needs to be processed before transmission, the prior art is limited by the design thereof, and a high-availability guarantee mechanism for an application or an application gateway is not available, which may cause a significant impact on the availability of the application itself, a traffic acquisition device, and an application gateway device, and may cause the performance of an application system, a traffic acquisition device, and an application gateway to be seriously degraded or unavailable.

To sum up, at present, there are ways of collecting user behavior data, application transmission data, and application asset data that cannot be managed and controlled, the collection is complex and inflexible, the efficiency is low, and the maintenance cost is high, and there is no guarantee mechanism that is highly available to the application in the transmission process of the collected data, which may cause significant impact on the availability of the application system, the traffic collection device, and the application gateway where the data is collected, and may cause severe performance degradation or unavailability.

Therefore, it is necessary to design a new method to realize the manageable collection of user behavior data, application transmission data and application asset data, and the collection is more comprehensive, simple and flexible, the efficiency is high, the maintenance cost is low, and the method has a highly available guarantee mechanism during transmission, and avoids the performance degradation of the application system and the application gateway of data collection, thereby seriously affecting the use of the application system.

Disclosure of Invention

The invention aims to overcome the defects of the prior art and provides a data acquisition and transmission method, a data acquisition and transmission device, a computer device and a storage medium.

In order to achieve the purpose, the invention adopts the following technical scheme: a data acquisition and transmission method comprising:

acquiring an acquisition strategy from a management terminal;

setting an actual acquisition strategy by adopting a switch mode according to the acquisition strategy;

intercepting the access data by adopting a byte enhancement technology to obtain intercepted data;

acquiring user behavior data, application transmission data and application asset data of the intercepted data by combining a linkage mechanism according to the actual acquisition strategy so as to obtain target data;

caching the target data;

and performing guaranteed transmission on the target data.

The further technical scheme is as follows: the method for acquiring user behavior data, application transmission data and application asset data of intercepted data by combining a linkage mechanism according to the actual acquisition strategy to obtain target data comprises the following steps:

acquiring performance index data;

adjusting the actual acquisition strategy according to the performance index data and a linkage mechanism to obtain a linkage acquisition strategy;

and acquiring user behavior data, application transmission data and application asset data of the intercepted data according to the linkage acquisition strategy to obtain target data.

The further technical scheme is as follows: the adjusting the actual acquisition strategy according to the performance index data and the linkage mechanism to obtain a linkage acquisition strategy comprises:

and adjusting the data acquisition frequency and the single data acquisition quantity of the actual acquisition strategy or closing a mode of a switch for setting data acquisition based on application service according to the performance index data and the corresponding threshold value to obtain a linkage acquisition strategy.

The further technical scheme is as follows: the performance index data includes at least one of a CPU usage rate, a memory usage rate, and a hard disk usage rate.

The further technical scheme is as follows: the caching the target data comprises:

acquiring data storage configuration;

judging whether the data storage configuration is a memory storage configuration;

if the data storage configuration is a memory storage configuration, storing the target data based on a bounded circular queue mode;

and if the data storage configuration is not the memory storage configuration, allocating a storage space under a specified disk path, and storing the target data into the storage space.

The further technical scheme is as follows: the performing the guaranteed transmission on the target data comprises:

judging whether the communication service with the data receiver is normal or not;

if the communication service with the data receiver is normal, reading the target data from the memory or the hard disk and sending the target data to the data receiver, and judging whether the data storage configuration is updated or not;

if the data storage configuration is updated, restoring the updated data storage configuration;

if the communication service with the data receiver is abnormal, judging whether to allow the automatic updating of the data storage configuration;

if the data storage configuration is allowed to be automatically updated, the data storage configuration is updated to the file system.

The further technical scheme is as follows: if the communication service with the data receiver is abnormal, after judging whether to allow the automatic updating of the data storage configuration, the method further comprises the following steps:

and if the data storage configuration is not allowed to be automatically updated, discarding the target data.

The present invention also provides a data acquisition and transmission apparatus comprising:

the strategy acquisition unit is used for acquiring an acquisition strategy from the management terminal;

the actual strategy setting unit is used for setting an actual acquisition strategy by adopting a switch mode according to the acquisition strategy;

the intercepting unit is used for intercepting the access data by adopting a byte enhancement technology to obtain intercepted data;

the acquisition unit is used for acquiring user behavior data, application transmission data and application asset data of the intercepted data by combining a linkage mechanism according to the actual acquisition strategy so as to obtain target data;

the cache unit is used for caching the target data;

and the transmission unit is used for performing guaranteed transmission on the target data.

The invention also provides computer equipment which comprises a memory and a processor, wherein the memory is stored with a computer program, and the processor realizes the method when executing the computer program.

The invention also provides a storage medium storing a computer program which, when executed by a processor, is operable to carry out the method as described above.

Compared with the prior art, the invention has the beneficial effects that: according to the invention, the data acquisition strategy is set at the management terminal and updated to the application client terminal, the application client terminal generates an actual acquisition strategy by combining the acquisition strategy in a switch mode, the data of a specific interface is intercepted by adopting a byte code enhancement technology, and the actual acquisition strategy is combined with a linkage mechanism to acquire corresponding data from multiple angles such as performance and the like, so that the performance is prevented from being influenced in the data acquisition process, the user behavior data, the application transmission data and the application asset data can be acquired in a controlled manner, and the acquisition is more comprehensive, simple and flexible, the efficiency is high, and the maintenance cost is low; the collected target data is cached firstly and then is subjected to guaranteed transmission, a high-availability guarantee mechanism is provided during transmission, and performance degradation of the application client is avoided.

The invention is further described below with reference to the accompanying drawings and specific embodiments.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings needed to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.

Fig. 1 is a schematic view of an application scenario of a data acquisition and transmission method according to an embodiment of the present invention;

fig. 2 is a schematic flow chart of a data acquisition and transmission method according to an embodiment of the present invention;

FIG. 3 is a sub-flow diagram of a data collection and transmission method according to an embodiment of the present invention;

FIG. 4 is a sub-flow diagram of a data collection and transmission method according to an embodiment of the present invention;

FIG. 5 is a sub-flow diagram of a data collection and transmission method according to an embodiment of the present invention;

FIG. 6 is a schematic block diagram of a data acquisition and transmission apparatus provided by an embodiment of the present invention;

FIG. 7 is a schematic block diagram of an acquisition unit of a data acquisition and transmission apparatus provided in an embodiment of the present invention;

FIG. 8 is a schematic block diagram of a buffer unit of a data acquisition and transmission apparatus according to an embodiment of the present invention;

FIG. 9 is a schematic block diagram of a transmission unit of a data acquisition and transmission apparatus provided by an embodiment of the present invention;

FIG. 10 is a schematic block diagram of a computer device provided by an embodiment of the present invention.

Detailed Description

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

It will be understood that the terms "comprises" and/or "comprising," when used in this specification and the appended claims, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.

It is also to be understood that the terminology used in the description of the invention herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used in the specification of the present invention and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.

It should be further understood that the term "and/or" as used in this specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

Referring to fig. 1 and fig. 2, fig. 1 is a schematic view of an application scenario of a data acquisition and transmission method according to an embodiment of the present invention. Fig. 2 is a schematic flow chart of a data acquisition and transmission method according to an embodiment of the present invention. The data acquisition and transmission method is applied to an application server, an agent, namely an application client side, is installed on the application server, the application client side communicates with a management side, the management side is generally an independent server, the application client side is pushed to the application client side in real time through an acquisition strategy arranged in the management side, an actual acquisition strategy is set by combining an acquisition switch arranged in the application client side, when data initiated by a user through a user terminal passes through a designated interface of the application client side, the application client side intercepts the data by adopting a byte enhancement technology, acquires the intercepted data by adopting the actual acquisition strategy, caches the acquired data, and the application server performs guaranteed transmission on target data, wherein a data receiving side can be another server or the management platform itself and the like.

It should be noted that fig. 1 only illustrates one application server, and in the actual operation process, a plurality of application servers may be simultaneously executed.

Fig. 2 is a schematic flow chart of a data acquisition and transmission method according to an embodiment of the present invention. As shown in fig. 2, the method includes the following steps S110 to S160.

And S110, acquiring an acquisition strategy from a management end.

The acquisition strategy comprises at least one data acquisition mode of data acquisition based on user identification configuration, data acquisition based on application service configuration, data acquisition based on an application interface, data acquisition according to fields of the application interface, data acquisition based on configured frequency and data acquisition based on configured single data acquisition quantity.

The method has the advantages that the deployment and acquisition of the network flow and the application gateway are limited by specific network areas and deployment positions, and the acquisition range is limited, so that the deployment is carried out on the application service host in a client-side application client-side mode during the deployment, the deployment is simple, the acquisition range is not limited, and the application client side is uniformly configured and managed by the management side with uniform management; the management terminal can configure single or combined data acquisition strategies based on application service, user identification, application interfaces, application interface fields, acquisition frequency and data acquisition quantity to acquire user behavior data, application transmission data and application asset data, controllability of data acquisition is achieved, and the strategy of the application client terminal for data acquisition can be changed by changing the acquisition strategies at the management terminal, so that controllability of data acquisition is achieved. The data acquisition strategy can be rapidly configured, fixed point data acquisition and development are not needed, manpower is saved, the development period is greatly shortened, and the work and project efficiency is improved; the collection of user behavior data, application transmission data and application asset data of a buried point and a second switch is not needed, the operation maintenance of the application and the influence on an application system are reduced, and therefore the normal operation of the application can be guaranteed more. When in deployment, the application service host is deployed in a client mode, and a management end with unified management performs unified configuration management on the application client; the method has no influence on the architecture of an application system and no influence on the existing network architecture.

Specifically, the data acquisition mode for performing data acquisition based on the user identifier configuration refers to whether application transmission data is acquired based on the user identifier configuration, and specifies that a single user, multiple users or all users may or may not acquire user behavior data application transmission data and application asset data, where the user identifier includes an accessed user account, an accessed IP, an accessed mac, an accessed browser, and the like.

The data acquisition mode for acquiring data based on the application service configuration refers to whether application transmission data and application asset data are acquired based on the application service configuration, and it is specified that a single or multiple or all application services may or may not acquire the application transmission data.

The data acquisition mode for acquiring data based on the application interfaces refers to that whether the application transmission data and the application asset data are acquired or not is based on the application interfaces, and single or multiple or all application interfaces are designated to acquire the application transmission data or not.

The data acquisition mode for acquiring data according to the field of the application interface refers to whether the field configuration of the application interface acquires application transmission data, the control is mainly performed through the interface field on the interface name, and single or multiple or all application interface fields are specified to acquire the application transmission data or not.

The data acquisition mode for acquiring data based on the configured frequency is controlled by the time period for acquiring data.

The data acquisition mode for carrying out data acquisition based on the configured single data acquisition amount is mainly controlled by the number of data to be acquired.

Whether the application transmission data is collected or not is configured in any combination of the data collection modes, for example, behavior data of a certain user accessing a certain interface is allowed to be collected or behavior data of a specified or all users accessing a specified certain application service is not allowed to be collected; thereby forming an acquisition strategy.

And S120, setting an actual acquisition strategy by adopting a switch mode according to the acquisition strategy.

In this embodiment, the actual acquisition policy refers to a policy actually used for acquiring data, and combines the acquisition policy transmitted by the client and the set switch mode to form a plurality of data acquisition policies, and the data acquisition policies are sequentially executed according to the set priorities. Specifically, an actual acquisition strategy is set in an application client side in a switch mode according to the acquisition strategy.

In this embodiment, the switch mode includes at least one of a switch for setting data acquisition based on an application service, a switch for setting data acquisition based on a user identifier, a switch for setting data acquisition based on an application interface, and a switch for setting data acquisition based on a field of an application interface; the switch mode also includes a period threshold for data acquisition and a filtering rule based on the acquisition amount of single data.

Specifically, setting the mode of the switch for data acquisition based on the application service means that when the state is enabled, the application client opens the data acquisition function of the application service, which is a precondition that the acquisition function can normally work; when the state is closed, the application client closes the acquisition function of the application service, and all the acquisition functions of the application service are closed.

Setting a mode of a switch for data acquisition based on the user identifier means that the application client opens a data acquisition function based on the user identifier when the state is enabled; when the state is off, the application client side closes the data acquisition function based on the user identification, but other acquisition switches and data acquisition are not affected, and only data acquisition is not carried out based on the user identification any more.

Setting the mode of a switch for data acquisition based on the application interfaces means that when the state is enabled, data acquisition is performed on all the application interfaces by default, and acquisition or non-acquisition can be defined for data of a designated interface, wherein the designated interface can be a single interface or a plurality of interfaces or a mark of combining all interface names with acquisition or non-acquisition; when the state is off, the data acquisition function of the application interface based on the application service is closed by default, but data of a designated interface such as single or multiple or all interface names can be acquired in combination with acquired marks, and other acquisition switches and data acquisition are not affected at the moment.

Setting the mode of the switch for data acquisition based on the field of the application interface means that when the state is enabled, data acquisition is performed on all interfaces and all fields by default, and data acquisition is not performed on the designated interfaces and designated fields such as single or multiple or all interface names plus the field names of the corresponding interfaces and non-acquired marks; when the state is off, the data acquisition function based on the application interface field is turned off, but other acquisition switches and data acquisition are not affected, and only application transmission data is not acquired based on the application interface field any more, but the field names of the designated interfaces and designated fields, such as single or multiple or all interface names, plus the field names of the corresponding interfaces and the acquired marked data are acquired, and at the moment, other acquisition switches and data acquisition are not affected.

The period threshold value of data acquisition is to control the data to be acquired by counting the time period of the acquired data, and the acquired data main body is a single application service, an application interface and an interface field in the service; the filtering rule based on the acquisition amount of single data is used for filtering data number extraction of single application service, application interfaces and interface fields in the service, and the filtering is carried out in the manners of pre-data extraction, post-data extraction, random extraction, data extraction from a certain specific data point and the like.

The management terminal and the application client terminal establish a bidirectional communication channel, can send instructions to the application client terminal at any time, the application client terminal can execute according to the instructions and return execution results, and the application client terminal can also actively send information such as the state of the application client terminal to the management terminal through the bidirectional communication channel. During initial installation, after the application client and the management terminal establish communication, the management terminal can push acquisition strategies according to an application client group or a single application client, when the data acquisition strategies are updated, strategy pushing can be carried out in real time, and the application client executes different data acquisition logics according to different acquisition instructions in the acquisition strategies aiming at the received data strategies, so that the function of managing and controlling data acquisition is achieved.

And S130, intercepting the access data by adopting a byte enhancement technology to obtain intercepted data.

In this embodiment, the intercepted data refers to data passing through a specified interface of the application client.

Specifically, after the Java bytecode is generated, the generated Java bytecode is dynamically modified during the JVM loading according to rules defined in the Java bytecode, and the content of the enhanced function is added, so that access data is intercepted according to the content of the enhanced function, and intercepted data is obtained.

By using a bytecode enhancement technology, after the Java bytecode is generated, according to rules defined in the Java bytecode, the generated Java bytecode is dynamically modified when a JVM (Java Virtual Machine) loads, and a required field is added or a method function is added or a new class and interface are inherited to be implemented. The dynamically added fields, or method functions, or inheritance to realize new classes and interfaces are all enhanced functions, and the enhanced functions mainly refer to the functions of automatically extracting request parameters and return results of the method functions.

Before extracting the relevant data, it is first determined that the data of interest must pass through the method function of the class in the course of one access by the user. After determining the method functions, acquiring data before and after calling the method functions by using a bytecode enhancement technology, and acquiring an application system to be acquired according to actual requirements, where the method functions include Object first (method) and/or Object second (method);

firstly, extracting the main content of the method (string parameter) as a function of the name of the method name random string, such as sdfsdfsd (parameter); then, the method (string parameter) side is used

Finally, the data collected in bef or () and after () will be stored in a variable in the first object, which is enhanced. The value in this variable can then be passed to bef in first method (String parameter) or to the after method, which concatenates the user's data throughout the lifetime of an access.

In the extracted data, there are different types of data, user identification information such as user account, IP, mac, browser, etc., user access time, access result, access interface, access data, access times, etc., but these data are dispersed in different places in the user access process, and through the above method functions, these data can be gathered together and gathered to any data access party, that is, all data are intercepted and gathered by using bytecode enhancement technology.

When the access data stream and the interface of user access and task scheduling call the Agent passing through the application client, the application client intercepts the flowing data by using the byte code enhancement technology, and the transmission data obtained by the byte code enhancement technology and analyzed by the application layer protocol is also acquired, so that the protocol does not need to be analyzed and the content is not restored, and the data is not influenced by the encryption protocol and the private protocol when being acquired. The collection of user behavior data, application transmission data and application asset data of a buried point and a second switch is not needed, the operation maintenance of the application and the influence on an application system are reduced, and the normal operation of the application can be further ensured; the method has no influence on the architecture of an application system and no influence on the existing network architecture.

The application client is deployed on the application server, so that the application components, the application host IP and the application examples can be directly acquired on an application system, and the application asset data can be acquired by combining the two data acquisition methods, namely the application name, the application service name, the application interface, the application components, the application host IP, the application examples, the database name, the database IP, the database port, the database type and the like.

And S140, acquiring user behavior data, application transmission data and application asset data of the intercepted data by combining the actual acquisition strategy with a linkage mechanism to obtain target data.

In this embodiment, the target data refers to a result obtained by screening and extracting user behavior data, application transmission data, and application asset data from the intercepted data by using a specific policy and mechanism.

In this embodiment, user behavior data, application transmission data and application asset data are acquired for intercepted data according to different data acquisition logics in the actual acquisition strategy and a preset linkage mechanism; wherein the different data acquisition logic performs data acquisition according to a set priority.

The user behavior data comprises user identification (a user account, an access IP, a mac, a browser and the like), acquisition time, access behaviors of the user, an access interface, the number of access times and the access data volume; the application transmission data comprises user identification, data transmission time, all protocol fields such as http and RPC (Remote Procedure Call), all contents of protocol fields such as http and RPC, all request data to be transmitted and all response data to be transmitted; the application asset data comprises a user identifier, an application name, an application service name, an application interface, an application component, an application host IP, an application instance, data transmission time, a database name, a database IP, a database port, a database category and the like;

in this embodiment, the priority is:

when a plurality of strategies are executed, processing is carried out according to the priority from high to low, and processing is carried out according to a data acquisition mode for carrying out data acquisition based on application service configuration, a data acquisition mode for carrying out data acquisition based on configured frequency, a data acquisition mode for carrying out data acquisition based on configured single data acquisition quantity, a data acquisition mode for carrying out data acquisition based on an application interface, a data acquisition mode for carrying out data acquisition according to fields of the application interface and a data acquisition mode for carrying out data acquisition based on user identification configuration.

When a plurality of strategies conflict, processing according to high priority, from high to low, according to a data acquisition mode of performing data acquisition based on application service configuration, a data acquisition mode of performing data acquisition based on configured frequency, a data acquisition mode of performing data acquisition based on configured single data acquisition amount, a data acquisition mode of performing data acquisition based on an application interface, a data acquisition mode of performing data acquisition according to fields of the application interface, and a data acquisition mode of performing data acquisition based on user identification configuration.

When the executed data acquisition strategy logic has a problem, the application client does not execute the data acquisition instruction and returns a failure result to the management terminal.

Compared with a preset buried point and a fixed point, the data which can be collected by the method is more comprehensive, and only simple configuration is needed during collection, so that collection of user behavior data, application transmission data and application asset data becomes simpler and more flexible.

In the data acquisition process, only user behavior data, application transmission data and application asset data required by a user are acquired by using an actual acquisition strategy during acquisition, and data acquisition is conducted with the actual requirements as guidance as far as possible; the linkage mechanism is used for acquiring data through a self-adaptive self-learning acquisition function.

In an embodiment, referring to fig. 3, the step S140 may include steps S141 to S143.

And S141, acquiring performance index data.

In this embodiment, the performance indicating data includes at least one of a CPU usage rate, a memory usage rate, and a hard disk usage rate.

And S142, adjusting the actual acquisition strategy according to the performance index data and the linkage mechanism to obtain a linkage acquisition strategy.

In this embodiment, the linkage collection policy refers to a data collection policy obtained by adjusting the data collection frequency, the single data collection amount, or the mode of closing the switch for setting data collection based on the application service of the actual collection policy according to the performance index data and the rule set by the linkage mechanism.

The linkage mechanism is formed by matching corresponding acquisition conditions according to threshold values of performance index data, and is mainly used for adjusting an actual acquisition strategy in real time, and in the process of data acquisition by combining the linkage mechanism and the actual acquisition strategy, performance influence and faults caused to applications can be avoided, excessive consumption of system resources of an application host is avoided, and normal service use provided by the applications is preferentially guaranteed.

Specifically, according to the performance index data and the corresponding threshold, adjusting the data acquisition frequency and the single data acquisition amount of the actual acquisition strategy or closing an application service acquisition switch to obtain a linkage acquisition strategy.

In this embodiment, a user can modify a linkage mechanism by using a front-end interactive configuration page of a client, and the user performs configuration customization of the linkage mechanism according to the self condition of the user, wherein the configuration customization comprises the self-defining of a threshold value of a performance index, and the self-defining provides functions of adding, editing and deleting; the linkage mechanism may be set according to the following rules:

rule one is as follows: when the CPU utilization rate is lower than 40%, or the memory utilization rate is lower than 40%, or the hard disk utilization rate is lower than 40%, data acquisition is carried out according to the existing actual acquisition strategy;

rule two: when the CPU utilization rate is 40% -50% or the memory utilization rate is 40% -50% or the hard disk utilization rate is 40% -50%, the acquisition frequency is reduced by 30% on the basis of the existing acquisition frequency set by an actual acquisition strategy, so that the acquisition frequency set in the past is 10 seconds, the acquisition frequency is changed into 5+5 × 30% to 6.5 seconds, namely, the acquisition frequency is changed into 13 seconds every 10 seconds in the past.

Rule three: when the CPU utilization rate is 50% -60% or the memory utilization rate is 50% -60% or the hard disk utilization rate is 50% -60%, on the basis of the existing acquisition frequency set by the actual acquisition strategy, the acquisition frequency is reduced by 50%, meanwhile, the single-time data acquisition amount set by the actual acquisition strategy is reduced by 50%, and the acquisition frequency set in the past is 10 seconds, then the acquisition frequency is changed into 10+10 × 30% to 13 seconds, namely, the acquisition frequency is changed into 13 seconds every 10 seconds in the past. When the single data collection amount is all, the data collection amount is reduced to 50%.

Rule four: and when the CPU utilization rate is more than 60 percent, or the memory utilization rate is more than 60 percent, or the hard disk utilization rate is more than 60 percent, directly closing the data acquisition function.

Of course, in other embodiments, the following steps may be performed according to different requirements and the rule of the configuration template customized linkage mechanism:

firstly, setting a rule name, customizing the rule name by user, and not repeating the rule name; then configuring a performance index threshold value, such as the configuration of CPU utilization rate, a user-defined operator and utilization rate percentage; the configuration of the memory utilization rate, the operator and the utilization rate percentage can be defined by user; the configuration of the hard disk utilization rate can define an operator and the utilization rate percentage by user; operators that can be used are: equal to, greater than and equal to, less than and equal to, and range values. Then, setting processing logic, namely performing self-defined processing logic setting based on three performance indexes of the CPU utilization rate, the memory utilization rate and the hard disk utilization rate, wherein the usable processing logic is AND or not, and performing logic setting based on two of the processing logic, such as the CPU utilization rate or the memory utilization rate and the hard disk utilization rate, wherein when one of the CPU utilization rate or the memory utilization rate reaches a threshold value and the hard disk utilization rate reaches the threshold value, a linkage mechanism is automatically started; and then, setting a linkage mechanism, wherein the acquisition frequency, the single data acquisition amount or the switch mode can be set singly or can be set completely, the default is in an unopened state, and all the settings are calculated based on the currently configured actual acquisition strategy. The data acquisition frequency setting may include decreasing the percentage of acquisition frequency or increasing the percentage of acquisition frequency or not enabling; the single data acquisition volume setting comprises a decrease percentage setting or an increase percentage setting or no activation; the acquisition switch mode can select 'on' or 'off' or 'not on'; after the above four configurations are completed, the configured rules can be saved. And finally, applying the configured rules to the application client, wherein the rules take effect after application aiming at single or multiple or all client applications.

Of course, the linkage mechanism can be determined by editing and deleting the preset linkage mechanism related rules.

And S143, acquiring user behavior data, application transmission data and application asset data of the intercepted data according to the linkage acquisition strategy to obtain target data.

The method comprises the following steps of adopting a linkage acquisition strategy, acquiring user behavior data, application transmission data and application asset data from multiple aspects such as an acquisition strategy, performance indexes, a switch and the like, deploying on an application service host in a client mode during deployment, and carrying out unified configuration management on application clients by a management end with unified management; according to the invention, a single or combined data acquisition strategy can be configured at a management end based on application service, user identification, an application interface field, acquisition frequency and data content to acquire user behavior data, application transmission data and application asset data, so that controllability of an acquisition process is realized, user behavior data, application transmission data and application asset data can be acquired in a controllable manner, and the acquisition is more comprehensive, simple and flexible, the efficiency is high, and the maintenance cost is low.

S150, caching the target data.

In this embodiment, after data is collected, the current application client caches the data first and then processes the data and then sends the data to the data receiver, instead of directly sending the data to the data receiver after collection, the data storage configuration needs to be read to determine whether the storage mode of the current data is memory storage or file system storage, then different data operations are performed according to different storage configurations, and under a default design, all the collected data are cached in the memory before being sent.

In an embodiment, referring to fig. 4, the step S150 may include steps S151 to S154.

And S151, acquiring data storage configuration.

In this embodiment, the data storage configuration refers to a memory storage configuration or a file system storage configuration. The memory storage configuration refers to storing data into a memory, and the file system storage configuration refers to storing data into a set disk.

S152, judging whether the data storage configuration is a memory storage configuration;

s153, if the data storage configuration is a memory storage configuration, storing the target data based on a bounded circular queue mode.

In this embodiment, when the data storage configuration is a memory storage configuration, the storage of the processing target data is performed based on a bounded circular queue, where the bounded circular queue refers to storing the acquired data in a circular queue of a memory, and the circular queue refers to allocating a fixed-size memory, and the size is configurable, which may avoid the exhaustion of memory resources of the application due to an excessively large amount of acquired data, and when the circular queue of the memory is running, when the allocable memory is used up by continuously inputting the acquired data, the subsequent data may cover the previous data, and such a cycle is performed, so that the acquired data is not retained all the time during acquisition, and the memory resources of the application host are not continuously occupied, thereby ensuring the availability of the application layer.

S154, if the data storage configuration is not the memory storage configuration, allocating a storage space under a specified disk path, and storing the target data into the storage space.

When the data storage is configured as file system storage, a fixed size block of storage space is allocated under a specified disk path, and this size is configurable. Before the target data is stored in the disk, whether the space of the currently allocable disk is enough for storing the target data is judged, if the allocated disk space is used up, the collected target data is discarded, the space of the disk is not used again, the disk is not filled with continuous data and is fully loaded, and therefore the usability of the application host on the disk capacity is guaranteed.

And S160, performing guaranteed transmission on the target data.

In this embodiment, when a data receiving side initiates a data transmission request to an application client, the application client monitors a memory queue and a storage path of a file in real time, and when target data is stored, the application client sends the target data to a back-end service of the data receiving side, but before sending, whether communication between the application client and the data receiving side is normal or not is judged.

In an embodiment, referring to fig. 5, the step S160 may include steps S161 to S166.

S161, judging whether the communication service with the data receiver is normal;

s162, if the communication service with the data receiver is normal, reading the target data from the memory or the hard disk and sending the target data to the data receiver, and judging whether the data storage configuration is updated or not;

if the data storage configuration is not updated, entering an end step;

and S163, if the data storage configuration is updated, restoring the updated data storage configuration.

If the communication service with the data receiver is normal, the application client will take the target data from the memory or the hard disk to the data receiver at the moment, and will actively check whether the data storage configuration is updated, if so, the updated storage configuration will be restored, and whether the data storage configuration is updated refers to whether the data storage configuration is updated from the memory storage to the file system storage or from the file system storage to the memory storage.

S164, if the communication service with the data receiver is abnormal, judging whether to allow the automatic updating of the data storage configuration;

s165, if the data storage configuration is allowed to be automatically updated, updating the data storage configuration into a file system;

and S166, if the data storage configuration is not allowed to be automatically updated, discarding the target data.

If the communication service with the data receiver is abnormal, a data storage configuration is read, and the data storage configuration indicates whether to allow the application to automatically switch to the configuration of the file system storage to ensure that the data is not lost when the communication service with the data receiver is abnormal. If the communication is allowed, the application client will actively update the storage mode of the data to be the file system, so that the subsequently generated target data is stored to be the file system until the initial configuration is restored after the communication is recovered and the normal data is sent. When not allowed, the target data is discarded to avoid the target data occupying space, thereby ensuring the overall guaranteed transmission.

According to the data acquisition and transmission method, the data acquisition strategy is set at the management end and is updated to the application client, the application client generates the actual acquisition strategy by combining the acquisition strategy in a switch mode, the data of a specific interface is intercepted by adopting a byte code enhancement technology, and the corresponding data is acquired from multiple angles such as performance by adopting an actual acquisition strategy combined linkage mechanism, so that the performance is prevented from being influenced in the data acquisition process, the user behavior data, the transmission data and the application asset data can be acquired in a control manner, and the data acquisition is more comprehensive, simple and flexible, the efficiency is high, and the maintenance cost is low; the collected target data is cached firstly and then is subjected to guaranteed transmission, a high-availability guarantee mechanism is provided during transmission, the performance reduction of an application client is avoided, and the performance reduction of an application system and an application gateway for data collection is avoided, so that the use of the application system is seriously influenced.

Fig. 6 is a schematic block diagram of a data acquisition and transmission apparatus 300 according to an embodiment of the present invention. As shown in fig. 6, the present invention also provides a data acquisition and transmission apparatus 300 corresponding to the above data acquisition and transmission method. The data acquisition and transmission apparatus 300 includes a unit for performing the above-described data acquisition and transmission method, and the apparatus may be configured in a server. Specifically, referring to fig. 6, the data acquisition and transmission apparatus 300 includes a policy obtaining unit 301, an actual policy setting unit 302, an intercepting unit 303, an acquisition unit 304, a buffer unit 305, and a transmission unit 306.

A policy obtaining unit 301, configured to obtain an acquisition policy from a management end; an actual policy setting unit 302, configured to set an actual acquisition policy in a switch mode according to the acquisition policy; the intercepting unit 303 is configured to intercept the access data by using a byte enhancement technology to obtain intercepted data; the acquisition unit 304 is used for acquiring user behavior data, application transmission data and application asset data of the intercepted data by combining a linkage mechanism according to the actual acquisition strategy so as to obtain target data; a cache unit 305 configured to cache the target data; a transmission unit 306, configured to perform guaranteed transmission on the target data.

In an embodiment, as shown in fig. 7, the acquisition unit 304 includes an index obtaining subunit 3041, an adjusting subunit 3042, and a data acquisition subunit 3043.

An index obtaining subunit 3041 configured to obtain performance index data; an adjusting subunit 3042, configured to adjust the actual acquisition policy according to the performance index data and a linkage mechanism, so as to obtain a linkage acquisition policy; and the data acquisition subunit 3043 is configured to acquire user behavior data, application transmission data, and application asset data for the intercepted data according to the linkage acquisition policy, so as to obtain target data.

In an embodiment, the adjusting subunit 3042 is configured to adjust the data acquisition frequency, the single data acquisition amount, or the mode of closing the switch for setting data acquisition based on the application service according to the performance index data and the corresponding threshold, so as to obtain the linkage acquisition policy.

In an embodiment, as shown in fig. 8, the caching unit 305 includes a configuration obtaining subunit 3051, a configuration determining subunit 3052, a first storing subunit 3053, and a second storing subunit 3054.

A configuration acquisition subunit 3051, configured to acquire a data storage configuration; a configuration determining subunit 3052, configured to determine whether the data storage configuration is a memory storage configuration; a first storage subunit 3053, configured to, if the data storage configuration is a memory storage configuration, store the target data based on a bounded circular queue; a second storage subunit 3054, configured to, if the data storage configuration is not a memory storage configuration, allocate a storage space under a specified disk path, and store the target data in the storage space.

In one embodiment, as shown in fig. 9, the transmission unit 306 includes a service determination subunit 3061, a transmission subunit 3062, a restoration subunit 3063, an update determination subunit 3064, an update subunit 3065, and a discard subunit 3066.

A service judging subunit 3061, configured to judge whether the communication service with the data receiving side is normal; a sending subunit 3062, configured to, if the communication service with the data receiving party is normal, read the target data from the memory or the hard disk and send the target data to the data receiving party, and determine whether the data storage configuration is updated; a restore subunit 3063, configured to restore the updated data storage configuration if the data storage configuration is updated; an update judging subunit 3064, configured to judge whether to allow automatic update of the data storage configuration if the communication service with the data receiving party is abnormal; an update subunit 3065, configured to update the data storage configuration to a file system if the data storage configuration is allowed to be automatically updated; a discard subunit 3066, configured to discard the target data if the automatic updating of the data storage configuration is not allowed.

It should be noted that, as can be clearly understood by those skilled in the art, the specific implementation processes of the data acquisition and transmission apparatus 300 and each unit may refer to the corresponding descriptions in the foregoing method embodiments, and for convenience and brevity of description, no further description is provided herein.

The data acquisition and transmission device 300 may be implemented in the form of a computer program that can be run on a computer device as shown in fig. 10.

Referring to fig. 10, fig. 10 is a schematic block diagram of a computer device according to an embodiment of the present application. The computer device 500 may be a server.

Referring to fig. 10, the computer device 500 includes a processor 502, memory, and a network interface 505 connected by a system bus 501, where the memory may include a non-volatile storage medium 503 and an internal memory 504.

The non-volatile storage medium 503 may store an operating system 5031 and a computer program 5032. The computer programs 5032 include program instructions that, when executed, cause the processor 502 to perform a data acquisition and transmission method.

The processor 502 is used to provide computing and control capabilities to support the operation of the overall computer device 500.

The internal memory 504 provides an environment for the execution of the computer program 5032 in the non-volatile storage medium 503, and when the computer program 5032 is executed by the processor 502, the processor 502 can be caused to perform a data acquisition and transmission method.

The network interface 505 is used for network communication with other devices. Those skilled in the art will appreciate that the configuration shown in fig. 10 is a block diagram of only a portion of the configuration relevant to the present teachings and is not intended to limit the computing device 500 to which the present teachings may be applied, and that a particular computing device 500 may include more or less components than those shown, or may combine certain components, or have a different arrangement of components.

Wherein the processor 502 is configured to run the computer program 5032 stored in the memory to implement the following steps:

acquiring an acquisition strategy from a management terminal; setting an actual acquisition strategy by adopting a switch mode according to the acquisition strategy; intercepting the access data by adopting a byte enhancement technology to obtain intercepted data; acquiring user behavior data, application transmission data and application asset data of the intercepted data by combining a linkage mechanism according to the actual acquisition strategy so as to obtain target data; caching the target data; and performing guaranteed transmission on the target data.

In an embodiment, when the processor 502 implements the step of acquiring user behavior data, application transmission data, and application asset data for the intercepted data according to the actual acquisition policy in combination with the linkage mechanism to obtain the target data, the following steps are specifically implemented:

acquiring performance index data; adjusting the actual acquisition strategy according to the performance index data and a linkage mechanism to obtain a linkage acquisition strategy; and acquiring user behavior data, application transmission data and application asset data of the intercepted data according to the linkage acquisition strategy to obtain target data.

The performance index data includes at least one of a CPU utilization rate, a memory utilization rate, and a hard disk utilization rate.

In an embodiment, when the processor 502 implements the step of adjusting the actual acquisition policy according to the performance index data and the linkage mechanism to obtain the linkage acquisition policy, the following steps are specifically implemented:

In an embodiment, when the processor 502 implements the step of caching the target data, the following steps are specifically implemented:

acquiring data storage configuration; judging whether the data storage configuration is a memory storage configuration; if the data storage configuration is a memory storage configuration, storing the target data based on a bounded circular queue mode; and if the data storage configuration is not the memory storage configuration, allocating a storage space under a specified disk path, and storing the target data into the storage space.

In an embodiment, when the processor 502 implements the step of performing the guaranteed transmission on the target data, the following steps are specifically implemented:

judging whether the communication service with the data receiver is normal or not; if the communication service with the data receiver is normal, reading the target data from the memory or the hard disk and sending the target data to the data receiver, and judging whether the data storage configuration is updated or not; if the data storage configuration is updated, restoring the updated data storage configuration; if the communication service with the data receiver is abnormal, judging whether to allow the automatic updating of the data storage configuration; if the data storage configuration is allowed to be automatically updated, the data storage configuration is updated to the file system.

In an embodiment, after the processor 502 determines whether to allow the step of automatically updating the data storage configuration if the communication service with the data receiving party is not normal, the following steps are further implemented:

It should be understood that in the embodiment of the present Application, the processor 502 may be a Central Processing Unit (CPU), and the processor 502 may also be other general purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field Programmable Gate Arrays (FPGAs) or other Programmable logic devices, discrete Gate or transistor logic devices, discrete hardware components, etc. Wherein a general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.

It will be understood by those skilled in the art that all or part of the flow of the method implementing the above embodiments may be implemented by a computer program instructing associated hardware. The computer program includes program instructions, and the computer program may be stored in a storage medium, which is a computer-readable storage medium. The program instructions are executed by at least one processor in the computer system to implement the flow steps of the embodiments of the method described above.

Accordingly, the present invention also provides a storage medium. The storage medium may be a computer-readable storage medium. The storage medium stores a computer program, wherein the computer program, when executed by a processor, causes the processor to perform the steps of:

In an embodiment, when the processor executes the computer program to acquire user behavior data, application transmission data, and application asset data of the intercepted data by combining the actual acquisition policy with the linkage mechanism to obtain the target data, the following steps are specifically implemented:

In an embodiment, when the processor executes the computer program to adjust the actual acquisition policy according to the performance index data and the linkage mechanism to obtain the linkage acquisition policy, the following steps are specifically implemented:

In an embodiment, when the processor executes the computer program to implement the step of caching the target data, the following steps are specifically implemented:

In an embodiment, when the processor executes the computer program to implement the step of performing the guaranteed transmission on the target data, the following steps are specifically implemented:

In an embodiment, after the processor executes the computer program to determine whether to allow the step of automatically updating the data storage configuration if the communication service with the data receiver is not normal, the processor further performs the following steps:

The storage medium may be a usb disk, a removable hard disk, a Read-Only memory (ROM) or a Read-Only memory (ROM), a magnetic disk, or an optical disk, and may store various computer readable storage media.

Those of ordinary skill in the art will appreciate that the elements and algorithm steps of the examples described in connection with the embodiments disclosed herein may be embodied in electronic hardware, computer software, or combinations of both, and that the components and steps of the examples have been described in a functional general in the foregoing description for the purpose of illustrating clearly the interchangeability of hardware and software. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.

In the embodiments provided in the present invention, it should be understood that the disclosed apparatus and method may be implemented in other ways. For example, the above-described apparatus embodiments are merely illustrative. For example, the division of each unit is only one logic function division, and there may be another division manner in actual implementation. For example, various elements or components may be combined or may be integrated into another system, or some features may be omitted, or not implemented.

The steps in the method of the embodiment of the invention can be sequentially adjusted, combined and deleted according to actual needs. The units in the device of the embodiment of the invention can be merged, divided and deleted according to actual needs. In addition, functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit.

The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a storage medium. Based on such understanding, the technical solution of the present invention essentially or partially contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a terminal, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention.

While the invention has been described with reference to specific embodiments, the invention is not limited thereto, and various equivalent modifications and substitutions can be easily made by those skilled in the art within the technical scope of the invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A method of data acquisition and transmission, comprising:

acquiring an acquisition strategy from a management terminal;

caching the target data;

and performing guaranteed transmission on the target data.

2. The data acquisition and transmission method according to claim 1, wherein the acquiring user behavior data, application transmission data and application asset data for the intercepted data according to the actual acquisition policy in combination with a linkage mechanism to obtain target data comprises:

acquiring performance index data;

3. The data collection and transmission method of claim 2, wherein the adjusting the actual collection strategy according to the performance index data and a linkage mechanism to obtain a linkage collection strategy comprises:

4. The data collection and transmission method of claim 2, wherein the performance indicator data comprises at least one of CPU usage, memory usage, and hard disk usage.

5. The data acquisition and transmission method of claim 1, wherein the buffering the target data comprises:

acquiring data storage configuration;

6. The data collection and transmission method of claim 1, wherein the guaranteed transmission of the target data comprises:

7. The data collection and transmission method of claim 6, wherein after determining whether to allow automatic updating of the data storage configuration if the communication service with the data receiver is not normal, further comprising:

8. Data acquisition and transmission apparatus, characterized in that it comprises:

the cache unit is used for caching the target data;

9. A computer device, characterized in that the computer device comprises a memory, on which a computer program is stored, and a processor, which when executing the computer program implements the method according to any of claims 1 to 7.

10. A storage medium, characterized in that the storage medium stores a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 7.