CN113079048A - Data acquisition method and device, computer equipment and storage medium - Google Patents
Data acquisition method and device, computer equipment and storage medium Download PDFInfo
- Publication number
- CN113079048A CN113079048A CN202110344846.7A CN202110344846A CN113079048A CN 113079048 A CN113079048 A CN 113079048A CN 202110344846 A CN202110344846 A CN 202110344846A CN 113079048 A CN113079048 A CN 113079048A
- Authority
- CN
- China
- Prior art keywords
- data
- application
- acquisition
- access
- strategy
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/08—Configuration management of networks or network elements
- H04L41/0803—Configuration setting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F8/00—Arrangements for software engineering
- G06F8/60—Software deployment
- G06F8/65—Updates
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/08—Configuration management of networks or network elements
- H04L41/0803—Configuration setting
- H04L41/0813—Configuration setting characterised by the conditions triggering a change of settings
- H04L41/082—Configuration setting characterised by the conditions triggering a change of settings the condition being updates or upgrades of network functionality
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/08—Configuration management of networks or network elements
- H04L41/0889—Techniques to speed-up the configuration process
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/08—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/14—Network architectures or network communication protocols for network security for detecting or protecting against malicious traffic
- H04L63/1408—Network architectures or network communication protocols for network security for detecting or protecting against malicious traffic by monitoring network traffic
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/20—Network architectures or network communication protocols for network security for managing network security; network security policies in general
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Computer Security & Cryptography (AREA)
- General Engineering & Computer Science (AREA)
- Computing Systems (AREA)
- Computer Hardware Design (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Environmental & Geological Engineering (AREA)
- Information Transfer Between Computers (AREA)
Abstract
The invention relates to a data acquisition method, a data acquisition device, computer equipment and a storage medium, wherein the method comprises the steps of acquiring an acquisition strategy from a management terminal; setting an actual acquisition strategy by adopting a switch mode according to the acquisition strategy; intercepting the access data by adopting a byte enhancement technology to obtain intercepted data; and acquiring user behavior data, transmission data and application asset data of the intercepted data according to the actual acquisition strategy. According to the invention, the data acquisition strategy is set at the management terminal and updated to the application client terminal, the application client terminal generates an actual acquisition strategy by adopting a switch mode and combining the acquisition strategy, the data of a specific interface is intercepted by adopting a byte code enhancement technology, and the acquisition of corresponding data is carried out by combining the actual acquisition strategy, so that the acquisition of user behavior data, transmission data and application asset data in a controllable manner is realized, and the acquisition is more comprehensive, simple and flexible, the efficiency is high, and the maintenance cost is low.
Description
Technical Field
The present invention relates to computers, and more particularly to a data acquisition method, apparatus, computer device, and storage medium.
Background
When network security and data security management and control are performed, big data real-time analysis, security analysis, traceability analysis, evidence obtaining and the like need to be performed on user behavior data and transmission data, so that scenes of user behavior data, application transmission data and application asset data are gradually increased, and the demand is increased.
At present, the acquisition of user behavior data, application transmission data and application asset data comprises three modes, one mode is to use a preset application program embedded point or a secondary development fixed point acquisition scheme to acquire the user behavior data and the application transmission data, but the mode needs secondary development aiming at each appointed acquisition point, the development and maintenance cost is extremely high, and the acquisition at each time needs secondary development, code modification and operation, so that the invasion exists to the original service; the switch for data acquisition needs to be maintained on codes, and interactive dynamic configuration and adjustment cannot be performed on data acquisition objects, such as only data of a designated user, a designated interface and a designated application service are acquired; the second way is to collect application transmission data and application asset data by analyzing network flow, which can not analyze the flow and data content of some private protocols, the network flow analysis can not analyze the flow and data analysis because private protocol fields and standards are not disclosed, and can not analyze the application interface and reduction transmission data of the application system, and can not analyze the encrypted flow, because the deployment mode is limited to the range of the collectable application transmission data, only can partially collect the data accessed by users, and can not collect the data call among application systems, the complexity of the network architecture can be improved, the difficulty of network operation and maintenance can be improved, the efficiency of fault elimination can be reduced, the reliability and usability of the whole network can be reduced, and the objects of data collection can not be managed and controlled by an interactive data collection strategy, such as by frequency of acquisition, data content, etc.; in another mode, application transmission data is acquired by erecting an application security gateway, and this mode cannot acquire applications not included in the application gateway range, and also can acquire only user access data included in the application gateway range, and cannot acquire data transmission between applications.
In summary, the current collection methods for user behavior data, application transmission data and application asset data cannot be managed and controlled, the collection is complex and inflexible, the efficiency is low, and the maintenance cost is high.
Therefore, it is necessary to design a new method for collecting user behavior data, application transmission data and application asset data in a controllable manner, and the collection is more comprehensive, simple and flexible, and has high efficiency and low maintenance cost.
Disclosure of Invention
The invention aims to overcome the defects of the prior art and provides a data acquisition method, a data acquisition device, computer equipment and a storage medium.
In order to achieve the purpose, the invention adopts the following technical scheme: a method of data acquisition comprising:
acquiring an acquisition strategy from a management terminal;
setting an actual acquisition strategy by adopting a switch mode according to the acquisition strategy;
intercepting the access data by adopting a byte enhancement technology to obtain intercepted data;
and acquiring user behavior data, application transmission data and application asset data for the intercepted data according to the actual acquisition strategy.
The further technical scheme is as follows: the acquisition strategy comprises at least one data acquisition mode of data acquisition based on user identification configuration, data acquisition based on application service configuration, data acquisition based on an application interface, data acquisition according to fields of the application interface, data acquisition based on configured frequency and data acquisition based on configured single data acquisition quantity.
The further technical scheme is as follows: the switch mode comprises at least one mode of a switch for setting data acquisition based on application service, a switch for setting data acquisition based on user identification, a switch for setting data acquisition based on an application interface and a switch for setting data acquisition based on a field of the application interface; the switch mode also includes a period threshold for data acquisition and a filtering rule based on the acquisition amount of single data.
The further technical scheme is as follows: the intercepting the access data by adopting the byte enhancement technology to obtain the intercepted data comprises the following steps:
after the Java bytecode is generated, dynamically modifying the generated Java bytecode during JVM loading according to rules defined in the Java bytecode, and adding the content of the enhanced function so as to intercept the access data according to the content of the enhanced function to obtain intercepted data.
The further technical scheme is as follows: the acquiring user behavior data, application transmission data and application asset data of intercepted data according to the actual acquisition strategy comprises the following steps:
acquiring user behavior data, application transmission data and application asset data for the intercepted data according to different data acquisition logics in the actual acquisition strategy; wherein the different data acquisition logic performs data acquisition according to a set priority.
The further technical scheme is as follows: the user behavior data comprises user identification, acquisition time, user access behavior, access interfaces, access data number and access data volume, and the user identification comprises a user account, an access IP, an access mac and an access browser; the application transmission data comprises user identification, data transmission time, all fields in the protocol, content corresponding to all fields in the protocol, all request data transmitted and all response data transmitted.
The further technical scheme is as follows: the application asset data includes an application name, an application service name, an application interface, an application component, an application host IP, an application instance, a database name, a database IP, a database port, and a database class.
The present invention also provides a data acquisition device comprising:
the strategy acquisition unit is used for acquiring an acquisition strategy from the management terminal;
the strategy setting unit is used for setting an actual acquisition strategy by adopting a switch mode according to the acquisition strategy;
the intercepting unit is used for intercepting the access data by adopting a byte enhancement technology to obtain intercepted data;
and the acquisition unit is used for acquiring user behavior data, application transmission data and application asset data of the intercepted data according to the actual acquisition strategy.
The invention also provides computer equipment which comprises a memory and a processor, wherein the memory is stored with a computer program, and the processor realizes the method when executing the computer program.
The invention also provides a storage medium storing a computer program which, when executed by a processor, is operable to carry out the method as described above.
Compared with the prior art, the invention has the beneficial effects that: according to the invention, the data acquisition strategy is set at the management terminal and updated to the application client terminal, the application client terminal generates an actual acquisition strategy by adopting a switch mode and combining the acquisition strategy, the data of a specific interface is intercepted by adopting a byte code enhancement technology, and the acquisition of corresponding data is carried out by combining the actual acquisition strategy, so that the acquisition of user behavior data, application transmission data and application asset data in a controllable manner is realized, and the acquisition is more comprehensive, simple and flexible, the efficiency is high, and the maintenance cost is low.
The invention is further described below with reference to the accompanying drawings and specific embodiments.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings needed to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.
Fig. 1 is a schematic view of an application scenario of a data acquisition method according to an embodiment of the present invention;
FIG. 2 is a schematic flow chart of a data acquisition method according to an embodiment of the present invention;
FIG. 3 is a schematic block diagram of a data acquisition device provided by an embodiment of the present invention;
FIG. 4 is a schematic block diagram of a computer device provided by an embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
It will be understood that the terms "comprises" and/or "comprising," when used in this specification and the appended claims, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.
It is also to be understood that the terminology used in the description of the invention herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used in the specification of the present invention and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.
It should be further understood that the term "and/or" as used in this specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.
Referring to fig. 1 and fig. 2, fig. 1 is a schematic view of an application scenario of a data acquisition method according to an embodiment of the present invention. Fig. 2 is a schematic flow chart of a data acquisition method according to an embodiment of the present invention. The data acquisition method is applied to an application server, an agent, namely an application client side, is installed on the application server, the application client side is communicated with a management side, the management side is an independent server, the application server is in data interaction with a user terminal, the application client side is pushed to the application client side in real time through an acquisition strategy arranged in the management side, an actual acquisition strategy is set by combining an acquisition switch arranged in the application client side, when data initiated by a user through the user terminal passes through a designated interface of the application client side, the application client side intercepts the data by adopting a byte enhancement technology, and acquires the intercepted data by adopting the actual acquisition strategy, so that the acquired data can be subsequently analyzed.
It should be noted that fig. 1 only illustrates one application server, and in the actual operation process, a plurality of application servers may be simultaneously executed.
Fig. 2 is a schematic flow chart of a data acquisition method according to an embodiment of the present invention. As shown in fig. 2, the method includes the following steps S110 to S140.
And S110, acquiring an acquisition strategy from a management end.
In this embodiment, the collection policy includes at least one data collection manner of data collection based on user identifier configuration, data collection based on application service configuration, data collection based on application interface, data collection according to a field of the application interface, data collection based on configured frequency, and data collection based on configured single data collection amount.
The management terminal can collect user behavior data, application transmission data and application asset data based on application service, user identification, application interface field, collection frequency and data collection quantity collection strategies, controllability of data collection is achieved, and the strategy of the application client for data collection can be changed by changing the collection strategies at the management terminal, so that controllability of data collection is achieved. The data acquisition strategy can be rapidly configured, fixed point data acquisition and development are not needed, manpower is saved, the development period is greatly shortened, and the work and project efficiency is improved; user behavior data, application transmission data and application asset data acquisition of a buried point and a second switch are not needed, operation maintenance of the application and influence on an application system are reduced, and therefore normal operation of the application can be guaranteed more. When in deployment, the application service host is deployed in a client mode, and a management end with unified management performs unified configuration management on the application client; the method has no influence on the architecture of an application system and no influence on the existing network architecture.
Specifically, the data acquisition mode for performing data acquisition based on the user identifier configuration refers to whether to acquire application transmission data based on the user identifier configuration, and specifies that a single user, multiple users or all users may or may not acquire user behavior data, application transmission data and application asset data, where the user identifier includes an accessed user account, an accessed IP, an accessed mac, an accessed browser, and the like.
The data acquisition mode for acquiring data based on application service configuration refers to whether user behavior data, application transmission data and application asset data are acquired based on application service configuration, and it is specified that single or multiple or all application services may or may not acquire application transmission data.
The data acquisition mode for acquiring data based on the application interface refers to that whether the application interface acquires user behavior data, application transmission data and application asset data or not, and a single or a plurality of or all application interfaces are designated to acquire the application transmission data or not.
The data acquisition mode for acquiring data according to the field of the application interface refers to whether field configuration of the application interface acquires user behavior data, application transmission data and application asset data, control is mainly performed through the interface field on the interface name, and single or multiple or all application interface fields are specified to acquire the application transmission data or not.
The data acquisition mode for acquiring data based on the configured frequency is controlled by the time period for acquiring data.
The data acquisition mode for carrying out data acquisition based on the configured single data acquisition amount is mainly controlled by the number of data to be acquired.
Whether the application transmission data is collected or not is configured in any combination of the data collection modes, for example, behavior data of a certain user accessing a certain interface is allowed to be collected or behavior data of a specified or all users accessing a specified certain application service is not allowed to be collected; thereby forming an acquisition strategy.
And S120, setting an actual acquisition strategy by adopting a switch mode according to the acquisition strategy.
In this embodiment, the actual acquisition policy refers to a policy actually used for acquiring data, and combines the acquisition policy transmitted by the client and the set switch mode to form a plurality of data acquisition policies, and the data acquisition policies are sequentially executed according to the set priorities.
In this embodiment, the switch mode includes at least one of a switch for setting data acquisition based on an application service, a switch for setting data acquisition based on a user identifier, a switch for setting data acquisition based on an application interface, and a switch for setting data acquisition based on a field of an application interface; the switch mode also includes a period threshold for data acquisition and a filtering rule based on the acquisition amount of single data.
Specifically, setting the mode of the switch for data acquisition based on the application service means that when the state is enabled, the application client opens the data acquisition function of the application service, which is a precondition that the acquisition function can normally work; when the state is closed, the application client closes the acquisition function of the application service, and all the acquisition functions of the application service are closed.
Setting a mode of a switch for data acquisition based on the user identifier means that the application client opens a data acquisition function based on the user identifier when the state is enabled; when the state is off, the application client side closes the data acquisition function based on the user identification, but other acquisition switches and data acquisition are not affected, and only data acquisition is not carried out based on the user identification any more.
Setting the mode of a switch for data acquisition based on the application interfaces means that when the state is enabled, data acquisition is performed on all the application interfaces by default, and acquisition or non-acquisition can be defined for data of a designated interface, wherein the designated interface can be a single interface or a plurality of interfaces or a mark of combining all interface names with acquisition or non-acquisition; when the state is off, the data acquisition function of the application interface based on the application service is closed by default, but data of a designated interface such as single or multiple or all interface names can be acquired in combination with acquired marks, and other acquisition switches and data acquisition are not affected at the moment.
Setting the mode of the switch for data acquisition based on the field of the application interface means that when the state is enabled, data acquisition is performed on all interfaces and all fields by default, and data acquisition is not performed on the designated interfaces and designated fields such as single or multiple or all interface names plus the field names of the corresponding interfaces and non-acquired marks; when the state is off, the data acquisition function based on the application interface field is turned off, but other acquisition switches and data acquisition are not affected, and only application transmission data is not acquired based on the application interface field any more, but the field names of the designated interfaces and designated fields, such as single or multiple or all interface names, plus the field names of the corresponding interfaces and the acquired marked data are acquired, and at the moment, other acquisition switches and data acquisition are not affected.
The period threshold value of data acquisition is to control the data to be acquired by counting the time period of the acquired data, and the acquired data main body is a single application service, an application interface and an interface field in the service; the filtering rule based on the acquisition amount of single data is used for filtering data number extraction of single application service, application interfaces and interface fields in the service, and the filtering is carried out in the manners of pre-data extraction, post-data extraction, random extraction, data extraction from a certain specific data point and the like.
The management terminal and the application client terminal establish a bidirectional communication channel, can send instructions to the application client terminal at any time, the application client terminal can execute according to the instructions and return execution results, and the application client terminal can also actively send information such as the state of the application client terminal to the management terminal through the bidirectional communication channel. During initial installation, after the application client and the management terminal establish communication, the management terminal can push acquisition strategies according to an application client group or a single application client, when the data acquisition strategies are updated, strategy pushing can be carried out in real time, and the application client executes different data acquisition logics according to different acquisition instructions in the acquisition strategies aiming at the received data strategies, so that the function of managing and controlling data acquisition is achieved.
And S130, intercepting the access data by adopting a byte enhancement technology to obtain intercepted data.
In this embodiment, the access data is data initiated by the user terminal; intercepted data refers to data passing through a designated interface of an application client.
Specifically, after the Java bytecode is generated, the generated Java bytecode is dynamically modified during the JVM loading according to rules defined in the Java bytecode, and the content of the enhanced function is added, so that access data is intercepted according to the content of the enhanced function, and intercepted data is obtained.
By using a bytecode enhancement technology, after the Java bytecode is generated, according to rules defined in the Java bytecode, the generated Java bytecode is dynamically modified when a JVM (Java Virtual Machine) loads, and a required field is added or a method function is added or a new class and interface are inherited to be implemented. The dynamically added fields, or method functions, or inheritance to realize new classes and interfaces are all enhanced functions, and the enhanced functions mainly refer to the functions of automatically extracting request parameters and return results of the method functions.
Before extracting the relevant data, it is first determined that the data of interest must pass through the method function of the class in the course of one access by the user. After determining the method functions, acquiring data before and after calling the method functions by using a bytecode enhancement technology, and acquiring an application system to be acquired according to actual requirements, where the method functions include Object first (method) and/or Object second (method);
first, the main content of the method (string parameter) is extracted as one partyThe function of the name of the normal name random string, such as sdfsdfsd (parameter); the method (string parameter) is then modified as follows:
finally, the data collected in before () and after () will be stored in a variable in the first object, and the variable is enhanced. The value in this variable is then passed to the before and after methods in first.
In the extracted data, there are different types of data, user identification information such as user account, IP, mac, browser, etc., user access time, access result, access interface, access data, access times, etc., but these data are dispersed in different places in the user access process, and through the above method functions, these data can be gathered together and gathered to any data access party, that is, all data are intercepted and gathered by using bytecode enhancement technology.
When the access data stream and the interface of user access and task scheduling call the Agent passing through the application client, the application client intercepts the flowing data by using the byte code enhancement technology, and the transmission data obtained by the byte code enhancement technology and analyzed by the application layer protocol is also acquired, so that the protocol does not need to be analyzed and the content is not restored, and the data is not influenced by the encryption protocol and the private protocol when being acquired. User behavior data, application transmission data and application asset data acquisition of a buried point and a second switch are not needed, operation maintenance of the application and the influence on an application system are reduced, and therefore normal operation of the application can be guaranteed more; the method has no influence on the architecture of an application system and no influence on the existing network architecture.
The application client is deployed on the application server, so that the application components, the application host IP and the application examples can be directly acquired on an application system, and the application asset data can be acquired by combining the two data acquisition methods, namely the application name, the application service name, the application interface, the application components, the application host IP, the application examples, the database name, the database IP, the database port, the database type and the like.
And S140, acquiring user behavior data, transmission data and application asset data of the intercepted data according to the actual acquisition strategy.
Specifically, the intercepted data stream is analyzed according to an actual acquisition strategy, and required user behavior data, application transmission data and application asset data are acquired.
In this embodiment, user behavior data, application transmission data, and application asset data are acquired for intercepted data according to different data acquisition logics within the actual acquisition policy; wherein the different data acquisition logic performs data acquisition according to a set priority.
In addition, the user behavior data comprises a user identifier, acquisition time, access behaviors of the user, an access interface, the number of access data and the amount of access data, wherein the user identifier comprises a user account, an access IP, an access mac and an access browser; the application transmission data comprises user identification, data transmission time, all fields in the protocol, contents corresponding to all fields in the protocol, all transmitted request data and all transmitted response data; the application transmission data includes a user identifier, a time of data transmission, all protocol fields such as http and RPC (Remote Procedure Call), all contents of protocol fields such as http and RPC (Remote Procedure Call), all request data to be transmitted, and all response data to be transmitted. The application asset data comprises a user identifier, an application name, an application service name, an application interface, an application component, an application host IP, an application instance, data transmission time, a database name, a database IP, a database port, a database category and the like.
In this embodiment, the priority is:
when a plurality of strategies are executed, processing is carried out according to the priority from high to low, and processing is carried out according to a data acquisition mode for carrying out data acquisition based on application service configuration, a data acquisition mode for carrying out data acquisition based on configured frequency, a data acquisition mode for carrying out data acquisition based on configured single data acquisition quantity, a data acquisition mode for carrying out data acquisition based on an application interface, a data acquisition mode for carrying out data acquisition according to fields of the application interface and a data acquisition mode for carrying out data acquisition based on user identification configuration.
When a plurality of strategies conflict, processing according to high priority, from high to low, according to a data acquisition mode of performing data acquisition based on application service configuration, a data acquisition mode of performing data acquisition based on configured frequency, a data acquisition mode of performing data acquisition based on configured single data acquisition amount, a data acquisition mode of performing data acquisition based on an application interface, a data acquisition mode of performing data acquisition according to fields of the application interface, and a data acquisition mode of performing data acquisition based on user identification configuration.
When the executed data acquisition strategy logic has a problem, the application client does not execute the data acquisition instruction and returns a failure result to the management terminal.
Compared with a preset buried point and a fixed point, the data which can be collected by the method is more comprehensive, and only simple configuration is needed during collection, so that collection of user behavior data, application transmission data and application asset data becomes simpler and more flexible.
According to the data acquisition method, the data acquisition strategy is set at the management terminal and updated to the application client, the application client generates the actual acquisition strategy by combining the acquisition strategy in a switch mode, the data of a specific interface is intercepted by adopting a byte code enhancement technology, and the corresponding data is acquired by combining the actual acquisition strategy, so that the user behavior data, the application transmission data and the application asset data can be acquired in a control manner, and the data acquisition is more comprehensive, simple and flexible, high in efficiency and low in maintenance cost.
Fig. 3 is a schematic block diagram of a data acquisition apparatus 300 according to an embodiment of the present invention. As shown in fig. 3, the present invention also provides a data collecting apparatus 300 corresponding to the above data collecting method. The data acquisition apparatus 300 includes means for performing the data acquisition method described above, and the apparatus may be in a server. Specifically, referring to fig. 3, the data acquisition apparatus 300 includes a policy obtaining unit 301, a policy setting unit 302, an intercepting unit 303, and an acquisition unit 304.
A policy obtaining unit 301, configured to obtain an acquisition policy from a management end; a policy setting unit 302, configured to set an actual acquisition policy in a switch mode according to the acquisition policy; the intercepting unit 303 is configured to intercept the access data by using a byte enhancement technology to obtain intercepted data; and the acquisition unit 304 is configured to acquire user behavior data, application transmission data, and application asset data from the intercepted data according to the actual acquisition policy.
In an embodiment, the intercepting unit 303 is configured to, after the Java bytecode is generated, dynamically modify the generated Java bytecode during the JVM loading according to rules defined in the Java bytecode, and add content of an enhanced function, so as to intercept access data according to the content of the enhanced function, so as to obtain intercepted data.
In an embodiment, the acquisition unit 304 is configured to acquire user behavior data, application transmission data, and application asset data for the intercepted data according to different data acquisition logics in the actual acquisition policy; wherein the different data acquisition logic performs data acquisition according to a set priority.
It should be noted that, as can be clearly understood by those skilled in the art, the specific implementation processes of the data acquisition apparatus 300 and each unit may refer to the corresponding descriptions in the foregoing method embodiments, and for convenience and brevity of description, no further description is provided herein.
The data acquisition apparatus 300 may be implemented in the form of a computer program that can be run on a computer device as shown in fig. 4.
Referring to fig. 4, fig. 4 is a schematic block diagram of a computer device according to an embodiment of the present application. The computer device 500 may be a server.
Referring to fig. 4, the computer device 500 includes a processor 502, memory, and a network interface 505 connected by a system bus 501, where the memory may include a non-volatile storage medium 503 and an internal memory 504.
The non-volatile storage medium 503 may store an operating system 5031 and a computer program 5032. The computer programs 5032 include program instructions that, when executed, cause the processor 502 to perform a data acquisition method.
The processor 502 is used to provide computing and control capabilities to support the operation of the overall computer device 500.
The internal memory 504 provides an environment for the operation of the computer program 5032 in the non-volatile storage medium 503, and when the computer program 5032 is executed by the processor 502, the processor 502 can be enabled to execute a data collection method.
The network interface 505 is used for network communication with other devices. Those skilled in the art will appreciate that the configuration shown in fig. 4 is a block diagram of only a portion of the configuration associated with the present application and does not constitute a limitation of the computer device 500 to which the present application may be applied, and that a particular computer device 500 may include more or less components than those shown, or may combine certain components, or have a different arrangement of components.
Wherein the processor 502 is configured to run the computer program 5032 stored in the memory to implement the following steps:
acquiring an acquisition strategy from a management terminal; setting an actual acquisition strategy by adopting a switch mode according to the acquisition strategy; intercepting the access data by adopting a byte enhancement technology to obtain intercepted data; and acquiring user behavior data, application transmission data and application asset data for the intercepted data according to the actual acquisition strategy.
The acquisition strategy comprises at least one data acquisition mode of data acquisition based on user identification configuration, data acquisition based on application service configuration, data acquisition based on an application interface, data acquisition according to fields of the application interface, data acquisition based on configured frequency and data acquisition based on configured single data acquisition amount.
The switch mode comprises at least one mode of a switch for setting data acquisition based on application service, a switch for setting data acquisition based on user identification, a switch for setting data acquisition based on an application interface and a switch for setting data acquisition based on a field of the application interface; the switch mode also includes a period threshold for data acquisition and a filtering rule based on the acquisition amount of single data.
In an embodiment, when implementing the step of intercepting the access data by using the byte enhancement technology to obtain the intercepted data, the processor 502 specifically implements the following steps:
after the Java bytecode is generated, dynamically modifying the generated Java bytecode during JVM loading according to rules defined in the Java bytecode, and adding the content of the enhanced function so as to intercept the access data according to the content of the enhanced function to obtain intercepted data.
In an embodiment, when the processor 502 implements the steps of collecting user behavior data, applying transmission data and applying asset data for intercepted data according to the actual collection policy, the following steps are specifically implemented:
acquiring user behavior data, application transmission data and application asset data for the intercepted data according to different data acquisition logics in the actual acquisition strategy; wherein the different data acquisition logic performs data acquisition according to a set priority.
The user behavior data comprises user identification, acquisition time, access behaviors of users, access interfaces, the number of access data and the amount of access data, and the user identification comprises a user account, an access IP, an access mac and an access browser; the application transmission data comprises user identification, data transmission time, all fields in the protocol, content corresponding to all fields in the protocol, all request data transmitted and all response data transmitted.
The application asset data includes an application name, an application service name, an application interface, an application component, an application host IP, an application instance, a database name, a database IP, a database port, and a database class.
It should be understood that in the embodiment of the present Application, the Processor 502 may be a Central Processing Unit (CPU), and the Processor 502 may also be other general-purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field Programmable Gate Arrays (FPGAs) or other Programmable logic devices, discrete Gate or transistor logic devices, discrete hardware components, and the like. Wherein a general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
It will be understood by those skilled in the art that all or part of the flow of the method implementing the above embodiments may be implemented by a computer program instructing associated hardware. The computer program includes program instructions, and the computer program may be stored in a storage medium, which is a computer-readable storage medium. The program instructions are executed by at least one processor in the computer system to implement the flow steps of the embodiments of the method described above.
Accordingly, the present invention also provides a storage medium. The storage medium may be a computer-readable storage medium. The storage medium stores a computer program, wherein the computer program, when executed by a processor, causes the processor to perform the steps of:
acquiring an acquisition strategy from a management terminal; setting an actual acquisition strategy by adopting a switch mode according to the acquisition strategy; intercepting the access data by adopting a byte enhancement technology to obtain intercepted data; and acquiring user behavior data, application transmission data and application asset data for the intercepted data according to the actual acquisition strategy.
The acquisition strategy comprises at least one data acquisition mode of data acquisition based on user identification configuration, data acquisition based on application service configuration, data acquisition based on an application interface, data acquisition according to fields of the application interface, data acquisition based on configured frequency and data acquisition based on configured single data acquisition amount.
The switch mode comprises at least one mode of a switch for setting data acquisition based on application service, a switch for setting data acquisition based on user identification, a switch for setting data acquisition based on an application interface and a switch for setting data acquisition based on a field of the application interface; the switch mode also includes a period threshold for data acquisition and a filtering rule based on the acquisition amount of single data.
In an embodiment, when the processor executes the computer program to implement the step of intercepting the access data by using the byte enhancement technology to obtain the intercepted data, the following steps are specifically implemented:
after the Java bytecode is generated, dynamically modifying the generated Java bytecode during JVM loading according to rules defined in the Java bytecode, and adding the content of the enhanced function so as to intercept the access data according to the content of the enhanced function to obtain intercepted data.
In an embodiment, when the step of acquiring user behavior data, application transmission data, and application asset data for the intercepted data according to the actual acquisition policy is implemented by executing the computer program, the processor specifically implements the following steps:
acquiring user behavior data, application transmission data and application asset data for the intercepted data according to different data acquisition logics in the actual acquisition strategy; wherein the different data acquisition logic performs data acquisition according to a set priority.
The user behavior data comprises user identification, acquisition time, access behaviors of users, access interfaces, the number of access data and the amount of access data, and the user identification comprises a user account, an access IP, an access mac and an access browser; the application transmission data comprises user identification, data transmission time, all fields in the protocol, content corresponding to all fields in the protocol, all request data transmitted and all response data transmitted. The application asset data includes an application name, an application service name, an application interface, an application component, an application host IP, an application instance, a database name, a database IP, a database port, and a database class. The storage medium may be a usb disk, a removable hard disk, a Read-Only Memory (ROM), a magnetic disk, or an optical disk, which can store various computer readable storage media.
Those of ordinary skill in the art will appreciate that the elements and algorithm steps of the examples described in connection with the embodiments disclosed herein may be embodied in electronic hardware, computer software, or combinations of both, and that the components and steps of the examples have been described in a functional general in the foregoing description for the purpose of illustrating clearly the interchangeability of hardware and software. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.
In the embodiments provided in the present invention, it should be understood that the disclosed apparatus and method may be implemented in other ways. For example, the above-described apparatus embodiments are merely illustrative. For example, the division of each unit is only one logic function division, and there may be another division manner in actual implementation. For example, various elements or components may be combined or may be integrated into another system, or some features may be omitted, or not implemented.
The steps in the method of the embodiment of the invention can be sequentially adjusted, combined and deleted according to actual needs. The units in the device of the embodiment of the invention can be merged, divided and deleted according to actual needs. In addition, functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit.
The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a storage medium. Based on such understanding, the technical solution of the present invention essentially or partially contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a terminal, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention.
While the invention has been described with reference to specific embodiments, the invention is not limited thereto, and various equivalent modifications and substitutions can be easily made by those skilled in the art within the technical scope of the invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.
Claims (10)
1. A data acquisition method, comprising:
acquiring an acquisition strategy from a management terminal;
setting an actual acquisition strategy by adopting a switch mode according to the acquisition strategy;
intercepting the access data by adopting a byte enhancement technology to obtain intercepted data;
and acquiring user behavior data, application transmission data and application asset data for the intercepted data according to the actual acquisition strategy.
2. The data collection method of claim 1, wherein the collection policy comprises at least one of data collection based on a user identification configuration, data collection based on an application service configuration, data collection based on an application interface, data collection based on a field of an application interface, data collection based on a configured frequency, and data collection based on a configured single data collection amount.
3. The data collection method of claim 1, wherein the switch mode comprises at least one of a switch for setting data collection based on an application service, a switch for setting data collection based on a user identifier, a switch for setting data collection based on an application interface, and a switch for setting data collection based on a field of an application interface; the switch mode also includes a period threshold for data acquisition and a filtering rule based on the acquisition amount of single data.
4. The data acquisition method of claim 1, wherein the intercepting access data by byte enhancement technology to obtain intercepted data comprises:
after the Java bytecode is generated, dynamically modifying the generated Java bytecode during JVM loading according to rules defined in the Java bytecode, and adding the content of the enhanced function so as to intercept the access data according to the content of the enhanced function to obtain intercepted data.
5. The data collection method of claim 1, wherein collecting user behavior data, application transmission data, and application asset data for the intercepted data according to the actual collection policy comprises:
acquiring user behavior data, application transmission data and application asset data for the intercepted data according to different data acquisition logics in the actual acquisition strategy; wherein the different data acquisition logic performs data acquisition according to a set priority.
6. The data acquisition method according to claim 1, wherein the user behavior data comprises user identification, acquisition time, user access behavior, access interface, number of access data pieces and access data amount, and the user identification comprises a user account, an access IP, an access mac and an access browser; the application transmission data comprises user identification, data transmission time, all fields in the protocol, content corresponding to all fields in the protocol, all request data transmitted and all response data transmitted.
7. The data collection method of claim 1, wherein the application asset data comprises an application name, an application service name, an application interface, an application component, an application host IP, an application instance, a database name, a database IP, a database port, and a database class.
8. Data acquisition device, its characterized in that includes:
the strategy acquisition unit is used for acquiring an acquisition strategy from the management terminal;
the strategy setting unit is used for setting an actual acquisition strategy by adopting a switch mode according to the acquisition strategy;
the intercepting unit is used for intercepting the access data by adopting a byte enhancement technology to obtain intercepted data;
and the acquisition unit is used for acquiring user behavior data, application transmission data and application asset data of the intercepted data according to the actual acquisition strategy.
9. A computer device, characterized in that the computer device comprises a memory, on which a computer program is stored, and a processor, which when executing the computer program implements the method according to any of claims 1 to 7.
10. A storage medium, characterized in that the storage medium stores a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110344846.7A CN113079048A (en) | 2021-03-30 | 2021-03-30 | Data acquisition method and device, computer equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110344846.7A CN113079048A (en) | 2021-03-30 | 2021-03-30 | Data acquisition method and device, computer equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113079048A true CN113079048A (en) | 2021-07-06 |
Family
ID=76611818
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110344846.7A Pending CN113079048A (en) | 2021-03-30 | 2021-03-30 | Data acquisition method and device, computer equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113079048A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114785616A (en) * | 2022-06-13 | 2022-07-22 | 深圳红途科技有限公司 | Data risk detection method and device, computer equipment and storage medium |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110083391A (en) * | 2019-03-16 | 2019-08-02 | 平安科技（深圳）有限公司 | Call request monitoring method, device, equipment and storage medium |
| CN110347557A (en) * | 2019-07-16 | 2019-10-18 | 深圳市网心科技有限公司 | Node resource monitoring method, electronic equipment, medium and acquisition method |
| CN111447170A (en) * | 2019-01-17 | 2020-07-24 | 北京京东尚科信息技术有限公司 | Data processing method and system, computer system and computer readable medium |
| CN111552521A (en) * | 2020-04-26 | 2020-08-18 | 深圳乐信软件技术有限公司 | Application data reporting method, device, server and storage medium |
-
2021
- 2021-03-30 CN CN202110344846.7A patent/CN113079048A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111447170A (en) * | 2019-01-17 | 2020-07-24 | 北京京东尚科信息技术有限公司 | Data processing method and system, computer system and computer readable medium |
| CN110083391A (en) * | 2019-03-16 | 2019-08-02 | 平安科技（深圳）有限公司 | Call request monitoring method, device, equipment and storage medium |
| CN110347557A (en) * | 2019-07-16 | 2019-10-18 | 深圳市网心科技有限公司 | Node resource monitoring method, electronic equipment, medium and acquisition method |
| CN111552521A (en) * | 2020-04-26 | 2020-08-18 | 深圳乐信软件技术有限公司 | Application data reporting method, device, server and storage medium |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114785616A (en) * | 2022-06-13 | 2022-07-22 | 深圳红途科技有限公司 | Data risk detection method and device, computer equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112738137B (en) | Data acquisition and link processing method and device, computer equipment and storage medium | |
| CN111565199B (en) | Network attack information processing method and device, electronic equipment and storage medium | |
| US10365985B2 (en) | Predictive management of on-demand code execution | |
| US9811363B1 (en) | Predictive management of on-demand code execution | |
| US9875353B2 (en) | Log information generation apparatus and recording medium, and log information extraction apparatus and recording medium | |
| US9830175B1 (en) | Predictive management of on-demand code execution | |
| US8271966B2 (en) | Methods and apparatus for patching software in accordance with instituted patching policies | |
| US20030204843A1 (en) | Automated installation of an application | |
| CN104239786B (en) | Exempt from ROOT Initiative Defenses collocation method and device | |
| KR20060042149A (en) | Method and system for filtering communications to prevent exploitation of a software vulnerability | |
| US10728267B2 (en) | Security system using transaction information collected from web application server or web server | |
| KR100875997B1 (en) | Network Fuzzing Using Windows Sockets Application Program Hooking | |
| WO2011060735A1 (en) | Method,device and system for invoking widget | |
| CN112231168A (en) | Micro server control method, device, equipment and storage medium | |
| CN106161396B (en) | A kind of method and device for realizing virtual machine network access control | |
| CN113067886B (en) | Database three-layer correlation auditing method and device, computer equipment and storage medium | |
| CN113079048A (en) | Data acquisition method and device, computer equipment and storage medium | |
| KR101658450B1 (en) | Security device using transaction information obtained from web application server and proper session id | |
| US20030204730A1 (en) | Secure transmission and installation of an application | |
| CN103034811A (en) | File processing method and system and device | |
| CN113055402B (en) | Data transmission control method and device, computer equipment and storage medium | |
| CN106936643B (en) | Equipment linkage method and terminal equipment | |
| CN115022008A (en) | Access risk assessment method, device, equipment and medium | |
| CN115086081A (en) | Escape prevention method and system for honeypots | |
| CN115883471A (en) | Application gateway and flow management and control method thereof |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB02 | Change of applicant information | ||
| CB02 | Change of applicant information |
Address after: Room 1706, 17 / F, Saixi science and technology building, No. 3398, Binhai Avenue, Binhai community, Yuehai street, Nanshan District, Shenzhen, Guangdong 518000 Applicant after: Shenzhen Hongtu Technology Co.,Ltd. Address before: Room 1706, 17 / F, Saixi science and technology building, No. 3398, Binhai Avenue, Binhai community, Yuehai street, Nanshan District, Shenzhen, Guangdong 518000 Applicant before: Shenzhen Hongtu chuangcheng Technology Co.,Ltd. |
|
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20210706 |