CN116185770A - Data acquisition method and device, electronic equipment and storage medium - Google Patents
Data acquisition method and device, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN116185770A CN116185770A CN202310084877.2A CN202310084877A CN116185770A CN 116185770 A CN116185770 A CN 116185770A CN 202310084877 A CN202310084877 A CN 202310084877A CN 116185770 A CN116185770 A CN 116185770A
- Authority
- CN
- China
- Prior art keywords
- log data
- host
- acquired
- index
- current
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3003—Monitoring arrangements specially adapted to the computing system or computing system component being monitored
- G06F11/3024—Monitoring arrangements specially adapted to the computing system or computing system component being monitored where the computing system component is a central processing unit [CPU]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3003—Monitoring arrangements specially adapted to the computing system or computing system component being monitored
- G06F11/3037—Monitoring arrangements specially adapted to the computing system or computing system component being monitored where the computing system component is a memory, e.g. virtual memory, cache
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3051—Monitoring arrangements for monitoring the configuration of the computing system or of the computing system component, e.g. monitoring the presence of processing resources, peripherals, I/O links, software programs
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3065—Monitoring arrangements determined by the means or processing involved in reporting the monitored data
- G06F11/3072—Monitoring arrangements determined by the means or processing involved in reporting the monitored data where the reporting involves data filtering, e.g. pattern matching, time or event triggered, adaptive or policy-based reporting
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Computing Systems (AREA)
- Quality & Reliability (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Mathematical Physics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Debugging And Monitoring (AREA)
- Storage Device Security (AREA)
Abstract
The present invention relates to the field of industrial network security technologies, and in particular, to a data acquisition method, a data acquisition device, an electronic device, and a storage medium. The method comprises the following steps: determining the acquisition sequence of a plurality of indexes to be acquired; for each index to be acquired, performing: inquiring and judging whether the current state of the host exceeds a threshold value, and if not, acquiring log data corresponding to the current index to be acquired; if yes, not collecting, and continuously inquiring and judging whether the current state of the host exceeds a threshold value in the next reporting period until the current state of the host is lower than the threshold value, and collecting log data corresponding to the index to be collected currently; the current state of the host comprises the current memory utilization rate and CPU utilization rate of the host; analyzing and preprocessing the collected log data to obtain preprocessed log data; and reporting the preprocessed log data to a preset server. The invention can adjust the data acquisition scheme according to the running state of the host computer, and ensure the safe running of the host computer.
Description
Technical Field
The present invention relates to the field of industrial network security technologies, and in particular, to a data acquisition method, a data acquisition device, an electronic device, and a storage medium.
Background
With the rapid development of information technology, information systems are widely and deeply applied, informatization construction is accelerated, and countries and industries and enterprises walk into big data times. The system is faced with information data brought by big data age, and increasingly hacking, and implementing national network security method and key information infrastructure network security protection requirements, perfecting network security supervision and management mechanism, and strengthening key construction of network security monitoring system is becoming more important.
The data acquisition is used as the first step of safety monitoring construction, and data information such as user behavior data, service support data, safety events and the like distributed in each service system is required to be acquired and stored, so that data support is provided for subsequent data audit, analysis, excavation, safety operation and the like. In the related art, data is acquired by utilizing the data proxy software, however, the intelligent degree of the existing data proxy software is low, the running state of the host cannot be judged autonomously in the data acquisition process, and judgment is needed by means of a downstream system, so that the host is halted, and the use of a user is influenced. Therefore, there is a need for a data acquisition method, apparatus, electronic device and storage medium to solve the above problems.
Disclosure of Invention
Based on the above problems, the invention provides a data acquisition method, a device, an electronic device and a storage medium, which can adjust a data acquisition scheme according to the running state of a host computer and ensure the safe running of the host computer.
In a first aspect, an embodiment of the present invention provides a data acquisition method, including:
determining the acquisition sequence of a plurality of indexes to be acquired;
for each index to be acquired, performing: inquiring and judging whether the current state of the host exceeds a threshold value, and if not, acquiring log data corresponding to the current index to be acquired; if yes, not collecting, and continuously inquiring and judging whether the current state of the host exceeds a threshold value in the next reporting period until the current state of the host is lower than the threshold value, and collecting log data corresponding to the index to be collected currently; the current state of the host comprises the current memory utilization rate and CPU utilization rate of the host;
analyzing and preprocessing the collected log data to obtain preprocessed log data;
and reporting the preprocessed log data to a preset server.
In one possible design, the determining whether the current state of the host exceeds a threshold includes:
inquiring and judging whether the current memory utilization rate of the host exceeds a memory threshold value, and/or
Inquiring and judging whether the current CPU utilization rate of the host exceeds a CPU threshold value.
In one possible design, the collecting log data corresponding to the currently to-be-collected index includes:
determining an acquisition mode for acquiring log data corresponding to the current index to be acquired based on the system version of the host;
based on the acquisition mode, acquiring log data corresponding to the current index to be acquired.
In one possible design, the analyzing and preprocessing the collected log data to obtain preprocessed log data includes:
determining an analysis mode of the collected log data based on the system version of the host and the collection mode of the log data;
analyzing the collected log data based on the analysis mode;
and filtering, cleaning, packaging, converting, mapping and escaping the analyzed log data by combining metadata management and data verification rules to obtain the preprocessed log data.
In one possible design, when the meaning of the parsed log data is incomplete, the method further includes:
and supplementing the analyzed log data by a method of data embedding points, basic information completion and associated fields.
In one possible design, the method further comprises obtaining basic information of the host.
In one possible design, the reporting the preprocessed log data to a preset server includes:
judging whether a preset reporting condition is met;
if yes, the basic information of the host is packaged into the preprocessed log data and is reported to a preset server together, and if not, the basic information of the host is not reported; the preset reporting condition comprises the running state of the host computer and the source of the log data when the log data are collected.
In a second aspect, an embodiment of the present invention further provides a data acquisition device, including:
the determining module is used for determining the acquisition sequence of a plurality of indexes to be acquired;
the acquisition module is used for acquiring log data corresponding to each index to be acquired, and the acquisition process is as follows: for each index to be acquired, performing: judging whether the current state of the host exceeds a threshold value, and if not, acquiring log data corresponding to the current index to be acquired; if yes, the log data corresponding to the index to be acquired is acquired, and whether the current state of the host exceeds a threshold value is continuously judged in the next reporting period until the current state of the host is lower than the threshold value; the current state of the host comprises the current memory utilization rate and CPU utilization rate of the host;
the processing module is used for analyzing and preprocessing the collected log data to obtain preprocessed log data;
and the reporting module is used for reporting the preprocessed log data to a preset server.
In a third aspect, an embodiment of the present invention further provides an electronic device, including a memory and a processor, where the memory stores a computer program, and when the processor executes the computer program, the method described in any embodiment of the present specification is implemented.
In a fourth aspect, embodiments of the present invention also provide a computer-readable storage medium having stored thereon a computer program which, when executed in a computer, causes the computer to perform a method according to any of the embodiments of the present specification.
In the embodiment of the invention, firstly, the acquisition sequence of a plurality of indexes to be acquired is determined, then, before each index to be acquired is acquired, whether the current state of the host exceeds a threshold value is firstly inquired and judged, if yes, the situation that the load of the host is excessive at the moment and the acquisition number possibly causes faults such as dead halt is indicated, and thus, the acquisition of data is suspended. Waiting for the next reporting period, continuously inquiring and judging whether the current state of the host exceeds a threshold value, if so, continuously waiting, and if not, collecting the corresponding log data. Therefore, the phenomenon that the host is dead due to overload caused by data collection can be avoided, and the normal operation of the host is ensured. After the log data is collected, the log data is analyzed and preprocessed, and then the log data can be reported to a preset server to prepare for subsequent data analysis. Therefore, the invention can adjust the data acquisition scheme according to the running state of the host computer, and ensure the safe running of the host computer.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings that are required in the embodiments or the description of the prior art will be briefly described, and it is obvious that the drawings in the following description are some embodiments of the present invention, and other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a flow chart of a data acquisition method according to an embodiment of the present invention;
FIG. 2 is a hardware architecture diagram of an electronic device according to an embodiment of the present invention;
fig. 3 is a block diagram of a data acquisition device according to an embodiment of the present invention.
Detailed Description
For the purpose of making the objects, technical solutions and advantages of the embodiments of the present invention more apparent, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention, and it is apparent that the described embodiments are some embodiments of the present invention, but not all embodiments, and all other embodiments obtained by those skilled in the art without making any inventive effort based on the embodiments of the present invention are within the scope of protection of the present invention.
Referring to fig. 1, an embodiment of the present invention provides a data acquisition method, which includes:
step 102, for each index to be collected, executing: inquiring and judging whether the current state of the host exceeds a threshold value, and if not, acquiring log data corresponding to the current index to be acquired; if yes, the log data corresponding to the index to be acquired is acquired, and the current state of the host is continuously inquired and judged in the next reporting period, and whether the current state of the host exceeds a threshold value or not is judged until the current state of the host is lower than the threshold value; the current state of the host comprises the current memory utilization rate and CPU utilization rate of the host;
104, analyzing and preprocessing the collected log data to obtain preprocessed log data;
and step 106, reporting the preprocessed log data to a preset server.
In the embodiment of the invention, firstly, the acquisition sequence of a plurality of indexes to be acquired is determined, then, before each index to be acquired is acquired, whether the current state of the host exceeds a threshold value is firstly inquired and judged, if yes, the situation that the load of the host is excessive at the moment and the acquisition number possibly causes faults such as dead halt is indicated, and thus, the acquisition of data is suspended. Waiting for the next reporting period, continuously inquiring and judging whether the current state of the host exceeds a threshold value, if so, continuously waiting, and if not, collecting the corresponding log data. Therefore, the phenomenon that the host is dead due to overload caused by data collection can be avoided, and the normal operation of the host is ensured. After the log data is collected, the log data is analyzed and preprocessed, and then the log data can be reported to a preset server to prepare for subsequent data analysis. Therefore, the invention can adjust the data acquisition scheme according to the running state of the host computer, and ensure the safe running of the host computer.
The implementation of the steps in fig. 1 is described in detail below.
First, for step 100, an acquisition order of a plurality of indicators to be acquired is determined.
The number of indexes to be collected can be determined according to the user requirement, and for example, the indexes can comprise behavior indexes (such as login time and login mode), service support indexes, safety event indexes and the like of the user, and after the collection sequence is determined, log data corresponding to each index can be collected item by item in a polling mode.
Next, for step 102, for each index to be collected, performing: judging whether the current state of the host exceeds a threshold value, and if not, acquiring log data corresponding to the current index to be acquired; if yes, the log data corresponding to the index to be acquired is acquired, and whether the current state of the host exceeds the threshold value is continuously judged in the next reporting period until the current state of the host is lower than the threshold value; the current state of the host includes the current memory usage and CPU usage of the host.
In this step, in order to ensure safe operation of the host, a memory threshold and a CPU threshold are set in advance. Then, before each index to be acquired is acquired, the current running state of the host can be automatically inquired, namely, the memory utilization rate or the CPU utilization rate of the host is inquired, whether the current memory utilization rate of the host exceeds a memory threshold value is inquired and judged, or whether the current CPU utilization rate of the host exceeds a CPU threshold value is inquired and judged. When at least one of the two main machine states exceeds a set threshold value, judging that the current state of the main machine exceeds the threshold value, and not collecting data at the moment so as to avoid faults such as dead halt caused by further increasing the load of the main machine. In this step, the memory threshold and the CPU threshold may take 90% and 95% respectively, and the values of the two may be the same or different, which is not specifically limited in this application.
And waiting for the next reporting period, wherein the reporting period is determined according to the user requirement, namely, every other reporting period, the running state inquiry of the host is carried out once to judge whether data acquisition is carried out or not. Therefore, the host machine can not be failed, the data collection can not be stopped for too long, and the data can be collected as comprehensively as possible under the condition that the safety of the host machine can be ensured.
In addition, the method for acquiring log data of the host computer by the same index item of different host computer systems is different, so in some embodiments, acquiring log data corresponding to the index to be acquired currently includes:
determining an acquisition mode for acquiring log data corresponding to the current index to be acquired based on a system version of the host;
based on the acquisition mode, acquiring log data corresponding to the current index to be acquired.
For example, for some host systems it is necessary to obtain log data by way of script, while for other systems it is necessary to obtain log data by way of code. According to the method, the proper acquisition mode can be automatically matched for each index item according to different host systems, the intervention is not required to be considered, and the data acquisition efficiency is improved.
Then, for step 104, the collected log data is parsed and preprocessed, so as to obtain preprocessed log data.
Similar to the above embodiment, since the log data formats of the same index item acquisition hosts of each host system are different, the parsed log data class is acquired according to the intelligent matching of each host system. Thus, step 104 includes:
determining an analysis mode of the collected log data based on a system version of the host and the collection mode of the log data;
analyzing the collected log data based on an analysis mode;
and filtering, cleaning, packaging, converting, mapping and escaping the analyzed log data by combining metadata management and data verification rules to obtain the preprocessed log data.
In the step, the collected log data is analyzed and preprocessed, so that impact of massive log data on a system can be avoided.
In some embodiments, when the meaning of the parsed log data is incomplete, the method further includes:
and supplementing the analyzed log data by the methods of data embedding points, basic information completion and associated fields so as to provide support for subsequent service processing.
In some embodiments, further comprising obtaining basic information of the host.
Finally, for step 106, the pre-processed log data is reported to a preset server, including:
judging whether a preset reporting condition is met;
if yes, packaging the basic information of the host into the preprocessed log data, and reporting the basic information to a preset server together, otherwise, not reporting the basic information; the preset reporting condition comprises the running state of the host computer when the log data are collected and the source of the log data.
In this step, the preset reporting condition may be set according to the user's requirement. For example, when the current memory usage rate or CPU usage rate of the host is lower than 60%, it indicates that the host is in good state, and the collected data may not be reported at this time; or E-disk data which is not concerned by the user is collected, the E-disk data can be not reported, so that more useless data can be filtered, useful data can be reserved, and further the load of computer analysis data is reduced. In addition, the basic information of the host is packaged into the preprocessed log data, so that the source of the data can be accurately distinguished when the data is utilized later.
Before data collection, the reporting IP, reporting port and reporting frequency of the data are set according to the need.
It should be further noted that the data acquisition method of the present invention is applicable to various host systems, such as windows system, linux system, and Unix system, and the like, and can be installed and used for the users of the above systems.
As shown in fig. 2 and 3, an embodiment of the present invention provides a data acquisition device. The apparatus embodiments may be implemented by software, or may be implemented by hardware or a combination of hardware and software. In terms of hardware, as shown in fig. 2, a hardware architecture diagram of an electronic device where a data acquisition device provided in an embodiment of the present invention is located, where the electronic device where the embodiment is located may include other hardware, such as a forwarding chip responsible for processing a packet, in addition to the processor, the memory, the network interface, and the nonvolatile memory shown in fig. 2. Taking a software implementation as an example, as shown in fig. 3, the device in a logic sense is formed by reading a corresponding computer program in a nonvolatile memory into a memory by a CPU of an electronic device where the device is located and running the computer program. The data acquisition device provided in this embodiment includes:
a determining module 300, configured to determine an acquisition order of a plurality of indicators to be acquired;
the collection module 302 is configured to collect log data corresponding to each index to be collected, where the collection process is: for each index to be acquired, performing: judging whether the current state of the host exceeds a threshold value, and if not, acquiring log data corresponding to the current index to be acquired; if yes, the log data corresponding to the index to be acquired is acquired, and whether the current state of the host exceeds the threshold value is continuously judged in the next reporting period until the current state of the host is lower than the threshold value; the current state of the host comprises the current memory utilization rate and CPU utilization rate of the host;
the processing module 304 is configured to parse and preprocess the collected log data to obtain preprocessed log data;
and the reporting module 306 is configured to report the preprocessed log data to a preset server.
In an embodiment of the present invention, the determining module 300 may be used to perform the step 100 in the above method embodiment, the collecting module 302 may be used to perform the step 102 in the above method embodiment, the processing module 304 may be used to perform the step 104 in the above method embodiment, and the reporting module 306 may be used to perform the step 106 in the above method embodiment.
In some embodiments, querying and determining whether the current state of the host exceeds a threshold includes:
inquiring and judging whether the current memory utilization rate of the host exceeds a memory threshold value, and/or
Inquiring and judging whether the current CPU utilization rate of the host exceeds a CPU threshold value.
In some embodiments, collecting log data corresponding to a current index to be collected includes:
determining an acquisition mode for acquiring log data corresponding to the current index to be acquired based on a system version of the host;
based on the acquisition mode, acquiring log data corresponding to the current index to be acquired.
In some implementations, the processing module 304 is configured to perform the following:
determining an analysis mode of the collected log data based on a system version of the host and the collection mode of the log data;
analyzing the collected log data based on an analysis mode;
and filtering, cleaning, packaging, converting, mapping and escaping the analyzed log data by combining metadata management and data verification rules to obtain the preprocessed log data.
In some embodiments, when the meaning of the parsed log data is incomplete, the method further includes:
and supplementing the analyzed log data by a method of data embedding points, basic information completion and associated fields.
In some embodiments, further comprising obtaining basic information of the host.
In some embodiments, the reporting module 306 is configured to perform the following operations:
judging whether a preset reporting condition is met;
if yes, packaging the basic information of the host into the preprocessed log data, and reporting the basic information to a preset server together, otherwise, not reporting the basic information; the preset reporting condition comprises the running state of the host computer when the log data are collected and the source of the log data.
It should be understood that the structure illustrated in the embodiments of the present invention is not limited to a specific type of data acquisition device. In other embodiments of the invention, a data acquisition device may include more or fewer components than shown, or certain components may be combined, or certain components may be split, or different arrangements of components. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.
The content of information interaction and execution process between the modules in the device is based on the same conception as the embodiment of the method of the present invention, and specific content can be referred to the description in the embodiment of the method of the present invention, which is not repeated here.
The embodiment of the invention also provides electronic equipment, which comprises a memory and a processor, wherein the memory stores a computer program, and the processor realizes the data acquisition method in any embodiment of the invention when executing the computer program.
The embodiment of the invention also provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program when executed by a processor causes the processor to execute a data acquisition method in any embodiment of the invention.
Specifically, a system or apparatus provided with a storage medium on which a software program code realizing the functions of any of the above embodiments is stored, and a computer (or CPU or MPU) of the system or apparatus may be caused to read out and execute the program code stored in the storage medium.
In this case, the program code itself read from the storage medium may realize the functions of any of the above-described embodiments, and thus the program code and the storage medium storing the program code form part of the present invention.
Examples of the storage medium for providing the program code include a floppy disk, a hard disk, a magneto-optical disk, an optical disk (e.g., CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD+RW), a magnetic tape, a nonvolatile memory card, and a ROM. Alternatively, the program code may be downloaded from a server computer by a communication network.
Further, it should be apparent that the functions of any of the above-described embodiments may be implemented not only by executing the program code read out by the computer, but also by causing an operating system or the like operating on the computer to perform part or all of the actual operations based on the instructions of the program code.
Further, it is understood that the program code read out by the storage medium is written into a memory provided in an expansion board inserted into a computer or into a memory provided in an expansion module connected to the computer, and then a CPU or the like mounted on the expansion board or the expansion module is caused to perform part and all of actual operations based on instructions of the program code, thereby realizing the functions of any of the above embodiments.
It is noted that relational terms such as first and second, and the like, are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus.
Those of ordinary skill in the art will appreciate that: all or part of the steps for implementing the above method embodiments may be implemented by hardware related to program instructions, and the foregoing program may be stored in a computer readable storage medium, where the program, when executed, performs steps including the above method embodiments; and the aforementioned storage medium includes: various media in which program code may be stored, such as ROM, RAM, magnetic or optical disks.
Finally, it should be noted that: the above embodiments are only for illustrating the technical solution of the present invention, and are not limiting; although the invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims (10)
1. A method of data acquisition, comprising:
determining the acquisition sequence of a plurality of indexes to be acquired;
for each index to be acquired, performing: inquiring and judging whether the current state of the host exceeds a threshold value, and if not, acquiring log data corresponding to the current index to be acquired; if yes, not collecting, and continuously inquiring and judging whether the current state of the host exceeds a threshold value in the next reporting period until the current state of the host is lower than the threshold value, and collecting log data corresponding to the index to be collected currently; the current state of the host comprises the current memory utilization rate and CPU utilization rate of the host;
analyzing and preprocessing the collected log data to obtain preprocessed log data;
and reporting the preprocessed log data to a preset server.
2. The method of claim 1, wherein querying and determining whether the current state of the host exceeds a threshold comprises:
inquiring and judging whether the current memory utilization rate of the host exceeds a memory threshold value, and/or
Inquiring and judging whether the current CPU utilization rate of the host exceeds a CPU threshold value.
3. The method according to claim 1, wherein the collecting log data corresponding to the currently to-be-collected index includes:
determining an acquisition mode for acquiring log data corresponding to the current index to be acquired based on the system version of the host;
based on the acquisition mode, acquiring log data corresponding to the current index to be acquired.
4. The method of claim 3, wherein the analyzing and preprocessing the collected log data to obtain preprocessed log data includes:
determining an analysis mode of the collected log data based on the system version of the host and the collection mode of the log data;
analyzing the collected log data based on the analysis mode;
and filtering, cleaning, packaging, converting, mapping and escaping the analyzed log data by combining metadata management and data verification rules to obtain the preprocessed log data.
5. The method of claim 4, further comprising, when the meaning of the parsed log data is incomplete:
and supplementing the analyzed log data by a method of data embedding points, basic information completion and associated fields.
6. The method of claim 1, further comprising obtaining basic information of the host.
7. The method of claim 6, wherein reporting the preprocessed log data to a preset server comprises:
judging whether a preset reporting condition is met;
if yes, the basic information of the host is packaged into the preprocessed log data and is reported to a preset server together, and if not, the basic information of the host is not reported; the preset reporting condition comprises the running state of the host computer and the source of the log data when the log data are collected.
8. A data acquisition device, comprising:
the determining module is used for determining the acquisition sequence of a plurality of indexes to be acquired;
the acquisition module is used for acquiring log data corresponding to each index to be acquired, and the acquisition process is as follows: for each index to be acquired, performing: inquiring and judging whether the current state of the host exceeds a threshold value, and if not, acquiring log data corresponding to the current index to be acquired; if yes, not collecting, and continuously inquiring and judging whether the current state of the host exceeds a threshold value in the next reporting period until the current state of the host is lower than the threshold value, and collecting log data corresponding to the index to be collected currently; the current state of the host comprises the current memory utilization rate and CPU utilization rate of the host;
the processing module is used for analyzing and preprocessing the collected log data to obtain preprocessed log data;
and the reporting module is used for reporting the preprocessed log data to a preset server.
9. An electronic device comprising a memory and a processor, the memory having stored therein a computer program, characterized in that the processor, when executing the computer program, implements the method of any of claims 1-7.
10. A storage medium having stored thereon a computer program, which, when executed in a computer, causes the computer to perform the method of any of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310084877.2A CN116185770A (en) | 2023-01-16 | 2023-01-16 | Data acquisition method and device, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310084877.2A CN116185770A (en) | 2023-01-16 | 2023-01-16 | Data acquisition method and device, electronic equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116185770A true CN116185770A (en) | 2023-05-30 |
Family
ID=86443813
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310084877.2A Pending CN116185770A (en) | 2023-01-16 | 2023-01-16 | Data acquisition method and device, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116185770A (en) |
-
2023
- 2023-01-16 CN CN202310084877.2A patent/CN116185770A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111866016B (en) | Log analysis method and system | |
| CN102713861B (en) | Operation management device, operation management method and program recorded medium | |
| CN111459782B (en) | Method and device for monitoring service system, cloud platform system and server | |
| CN113553210A (en) | Alarm data processing method, device, equipment and storage medium | |
| CN109379390B (en) | Network security baseline generation method based on full flow | |
| EP3623983A1 (en) | Method and device for identifying security threats, storage medium, processor and terminal | |
| CN112306802A (en) | Data acquisition method, device, medium and electronic equipment of system | |
| CN111224807A (en) | Distributed log processing method, device, equipment and computer storage medium | |
| CN107317708B (en) | Monitoring method and device for court business application system | |
| CN108763916B (en) | Service interface security assessment method and device | |
| US9201752B2 (en) | System and method for correlating empirical data with user experience | |
| US8949669B1 (en) | Error detection, correction and triage of a storage array errors | |
| CN107562555A (en) | The cleaning method and server of duplicate data | |
| JP5240709B2 (en) | Computer system, method and computer program for evaluating symptom | |
| CN116185770A (en) | Data acquisition method and device, electronic equipment and storage medium | |
| US8429458B2 (en) | Method and apparatus for system analysis | |
| CN107193721B (en) | Method and device for generating log | |
| KR101288535B1 (en) | Method for monitoring communication system and apparatus therefor | |
| CN110569172B (en) | Performance monitoring system of service level | |
| JP2019009726A (en) | Fault separating method and administrative server | |
| CN116432240B (en) | Method, device, server and system for detecting sensitive data of intranet terminal | |
| WO2024060245A1 (en) | Method and apparatus for analyzing device trust level, electronic device, and storage medium | |
| CN113596051B (en) | Detection method, detection apparatus, electronic device, medium, and computer program | |
| CN112527755B (en) | Government affair data exchange method and system based on block chain technology | |
| KR102623432B1 (en) | Apparatus and method for collecting meta information related to malicious code |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |