CN112612767A - Log file rapid analysis method and device - Google Patents
Log file rapid analysis method and device Download PDFInfo
- Publication number
- CN112612767A CN112612767A CN202011626206.7A CN202011626206A CN112612767A CN 112612767 A CN112612767 A CN 112612767A CN 202011626206 A CN202011626206 A CN 202011626206A CN 112612767 A CN112612767 A CN 112612767A
- Authority
- CN
- China
- Prior art keywords
- log file
- log
- regular
- analysis
- subfile
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/1805—Append-only file systems, e.g. using logs or journals to store data
- G06F16/1815—Journaling file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/16—File or folder operations, e.g. details of user interfaces specifically adapted to file systems
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Human Computer Interaction (AREA)
- Debugging And Monitoring (AREA)
Abstract
The invention relates to a method and a device for rapidly analyzing log files, which are applied to a distributed system, wherein the method comprises the following steps: acquiring and storing a regular analysis object set, and acquiring a log file to be analyzed; cutting the log file into a plurality of subfiles, and reading and loading each subfile; distributing the read and loaded subfiles to a plurality of processing units in parallel; each processing unit respectively matches the regular analysis object of the corresponding subfile in the regular analysis object set and analyzes the corresponding subfile; and performing warehousing operation on the analysis results of each processing unit. The invention has the technical effects of high log analysis efficiency, low consumption of the memory and the CPU and suitability for overlarge log files.
Description
Technical Field
The invention relates to the technical field of log analysis, in particular to a method and a device for rapidly analyzing log files and a computer storage medium.
Background
A large number of log files are generated in various devices and systems in various industries. The analysis of the log file can provide valuable and reliable index parameters for various devices and systems, and is used for supporting the operation, improvement and development of the various devices and systems. For example, banks, enterprises and government units all have a plurality of firewall devices, log data generated by each firewall device in the operation process are massive, and the logs are stored by taking days as units or categories as units to form log files; big data analysis is carried out based on the log files, multi-dimensional safety protection data with different granularities can be provided, and heavyweight data support is provided for network safety. In addition, a large amount of log data can be generated in the operation process of a business system or a production management system used by the traditional manufacturing industry, an e-commerce platform, a finance, logistics, aviation and social contact platform to form a log file; under the scene of big data analysis, the log files can dig out valuable data and generate various reliable indexes and reports which can be used for reference, so as to support the rapid development of later-stage business; useful data indexes such as bottlenecks, limitations, growing space of business development and the like can be found based on log analysis.
Currently, the log analysis may adopt a regular analysis scheme, and a scheme of implementing the log analysis by using xml configuration, and the like. Most of the schemes are not perfect enough, and when a large log file is analyzed, especially when a huge log file (the size of the log file exceeds 10G, the number of log file lines is more than or equal to 200 ten thousand) is analyzed, the analysis speed is too slow, and the consumption of a memory and a processor is large. Moreover, the memory occupied for a long time during analysis is not released, the CPU utilization rate is high, the running efficiency of other software is low, the response of an operating system is delayed, and the operation experience of a client is influenced; the conditions of blocking and data loss can occur under the condition that the log file is slightly larger, and the blocking condition is more serious when the overlarge log file is analyzed.
Disclosure of Invention
In view of the above, a method and an apparatus for fast parsing a log file are needed to solve the problems of low parsing efficiency of a large log file, high consumption of a memory and a processor, and easy jamming.
The invention provides a method for rapidly analyzing log files, which is applied to a distributed system and comprises the following steps:
acquiring and storing a regular analysis object set, and acquiring a log file to be analyzed;
cutting the log file into a plurality of subfiles, and reading and loading each subfile;
distributing the read and loaded subfiles to a plurality of processing units in parallel;
each processing unit respectively matches the regular analysis object of the corresponding subfile in the regular analysis object set and analyzes the corresponding subfile;
and performing warehousing operation on the analysis results of each processing unit.
Further, the method also comprises the following steps:
and after the analysis processing of all the subfiles of one log file is finished, setting time for dormancy, and then analyzing the next log file.
Further, acquiring and storing a regular analysis object set, specifically:
and setting a special storage space, receiving the regular analysis object set sent by the java component through an activeMq message queue, and storing the regular analysis object set to the special storage space.
Further, before the log file is divided into a plurality of subfiles and each subfile is read and loaded, the method further includes:
and judging whether the log file is a compressed file, if so, decompressing the log file, and then moving the decompressed log file to an analysis temporary directory, otherwise, directly moving the log file to the analysis temporary directory.
Further, the log file is divided into a plurality of subfiles, and each subfile is read and loaded, specifically:
judging whether the total line number of the log file is larger than a set line number, if so, cutting the log file into sub files with a plurality of set line numbers, otherwise, not cutting;
acquiring a path address of each subfile;
and reading each subfile in parallel or in series based on the path address, and loading the subfiles to a memory buffer area.
Further, the set number of lines and the size of the memory buffer area are set according to configuration parameters of the distributed system.
Further, distributing the read loaded subfiles to a plurality of processing units in parallel, further comprising:
and each processing unit distributes the log data in the corresponding sub-file to the peer processing unit or the next processing unit.
Further, each processing unit respectively matches the regular parsing object of the corresponding subfile in the regular parsing object set, and performs parsing on the corresponding subfile, specifically:
each processing unit preliminarily analyzes the corresponding subfile, acquires a log header ip, and matches a corresponding regular analysis object in the regular analysis object set through the log header ip;
each processing unit carries out analysis field matching on the corresponding subfile according to the regular analysis object obtained by matching, and if the matching is successful, analysis field data of the corresponding subfile are extracted;
and combining the analysis field data obtained by matching each processing unit to form an analysis result.
The invention also provides a device for rapidly analyzing the log file, which comprises a processor and a memory, wherein the memory is stored with a computer program, and the computer program is executed by the processor to realize the method for rapidly analyzing the log file.
The invention also provides a computer storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the method for rapidly analyzing the log file is realized.
Has the advantages that: before the log file is analyzed, the log file is cut, and then the cut sub-files are analyzed and processed by adopting a plurality of processing units. The log file is divided to improve the log analysis speed, and meanwhile, the log analysis process can be operated even in the environment with low CPU and low memory, so that the problems of system jamming and data loss caused by the analysis of the log file are avoided. Meanwhile, when each processing unit carries out log analysis, the adopted regular analysis object is obtained by real-time matching in the regular analysis object set, and the regular analysis object set is not configured in the distributed system but is obtained and configured through other components/systems except the distributed system.
Drawings
FIG. 1 is a flowchart of a log file fast parsing method according to a first embodiment of the present invention;
FIG. 2 is a schematic diagram of a regular parsing object set loading process according to a first embodiment of the log file fast parsing method provided by the present invention;
FIG. 3 is a flowchart illustrating a first embodiment of a method for fast parsing a log file according to the present invention;
FIG. 4 is a processing unit data flow chart of a log file fast parsing method according to a first embodiment of the present invention;
FIG. 5 is a schematic diagram of a regularized parsing process of a first embodiment of a fast log file parsing method provided by the present invention;
fig. 6 is a schematic view of a modular structure of a log file fast parsing apparatus according to a first embodiment of the present invention.
Detailed Description
The accompanying drawings, which are incorporated in and constitute a part of this application, illustrate preferred embodiments of the invention and together with the description, serve to explain the principles of the invention and not to limit the scope of the invention.
Example 1
As shown in fig. 1, an embodiment 1 of the present invention provides a method for quickly parsing a log file, which is hereinafter referred to as the method for short, and is applied to a distributed system, and includes the following steps:
s1, acquiring and storing a regular analysis object set, and acquiring a log file to be analyzed;
s2, cutting the log file into a plurality of subfiles, and reading and loading each subfile;
s3, distributing the read and loaded subfiles to a plurality of processing units in parallel;
s4, each processing unit respectively matches the regular analysis object of the corresponding subfile in the regular analysis object set, and analyzes the corresponding subfile;
and S5, performing warehousing operation on the analysis results of each processing unit.
The method for rapidly analyzing the log file is deployed in a distributed system, the distributed system is an Apache Storm which is a freely-open-source distributed real-time computing system in the embodiment, mass data is well processed, the method is suitable for data real-time processing rather than batch processing, various operations can be performed on the real-time data in parallel, the horizontal expansion is convenient and rapid, and higher analysis performance is obtained. Specifically, S101, a regular analysis object set is firstly obtained, so that implementation configuration of the regular analysis object is performed during subsequent log analysis; s102, finding out all log files needing to be analyzed; then cutting a single oversized log file into small files, namely subfiles, obtaining the path addresses of the cut subfiles, reading and loading each subfile in parallel or in series, wherein the size of a memory buffer area for loading the subfiles can be set according to the configuration of a distributed system so as to achieve optimal performance configuration; and distributing the read and loaded data to a plurality of distributed processing units of a distributed system in parallel, wherein each processing unit respectively acquires a corresponding regular analysis object to analyze a corresponding subfile, an analysis result set forms an sgn suffix file to be warehoused, and warehousing writing operation is carried out on the sgn suffix file.
In this embodiment, before parsing the log file, the log file is cut, and then the cut sub-files are parsed by using the plurality of processing units. The log analysis speed is improved by dividing the log file and can reach 15 ten thousand per second; meanwhile, the log analysis process can be operated even in the environment with low CPU and low memory, especially for the ultra-large log file, the processing mode can greatly improve the log analysis efficiency, the ultra-large log file can be rapidly and accurately read and analyzed under the condition of low memory and low CPU consumption, and the problems of jamming and data loss of the system caused by the analysis of the log file are avoided. Meanwhile, when each processing unit carries out log analysis, the adopted regular analysis object is obtained by real-time matching in the regular analysis object set, and the regular analysis object set is not configured in the distributed system but is obtained and configured through other components/systems except the distributed system, so that the regular analysis and the distributed system are decoupled, and the low-coupling design enables the analysis process to have higher expansibility and facilitates later-stage service expansion.
The method for rapidly analyzing the log file provided by the embodiment is suitable for analyzing the log file with any size, but is particularly suitable for the huge log file because the analysis efficiency is high and the consumption on the memory and the CPU is low.
Preferably, the method further comprises the following steps:
and after the analysis processing of all the subfiles of one log file is finished, setting time for dormancy, and then analyzing the next log file.
On the basis of log file cutting and regular analysis object low coupling configuration, the embodiment also sets sleep time, for example, sleep for 1 millisecond after completing an analysis process of a complete log file, so that the memory and the CPU are released, the use and occupation of the CPU and the memory can be further reduced instantly, and low operating efficiency of other software, delay of an operating system and influence on customer operation experience due to the fact that the analysis of the log file occupies the memory for a long time and the CPU has high utilization rate and is not released are avoided. Especially for the super large log file, the method can rapidly read and analyze the super large log file data without damage, with low CPU utilization rate and low memory consumption, and is light in weight, low in coupling and rapid in adapting to more log formats.
Preferably, the acquiring and storing a regular analysis object set specifically includes:
and setting a special storage space, receiving the regular analysis object set sent by the java component through an activeMq message queue, and storing the regular analysis object set to the special storage space.
Specifically, as shown in fig. 2, in this embodiment, the regular parsing object set is configured in a java component, the java component is a component at storm level, and is a function item component specially set for implementing log parsing, and in order to decouple design and facilitate client operation, an association binding relationship between a regular configuration file and a vendor log format is implemented in the java component, and the work of the regular configuration parsing file can be implemented in any java item component. In this embodiment, the java component that implements the association binding between the regular parsing configuration file and the vendor log format is a risk component, and as shown in fig. 2, the process of performing the regular parsing on the object set by the risk component specifically includes: the mapping relation setting unit in the risk component is used for setting a mapping relation between equipment ip (the equipment ip is used for describing different log formats) and the regular configuration analysis file, and binding the equipment ip and the regular configuration analysis file is realized; a regular parsing object set loading unit in the risk component is used for loading all regular configuration parsing files and converting the regular configuration parsing files into a regular parsing object set; in this embodiment, the regular analysis object set is a hash set of regular analysis objects, which facilitates matching of the regular analysis objects by using a hash matching method in the subsequent analysis. When the distributed system analyzes the log file, the regular expression analysis object set sent by the java component is received through the activeMq message queue, and a special storage space is arranged in the distributed system and used for storing the regular analysis object set and is directly used when a subsequent log processing unit analyzes the log.
The low-coupling structure setting of the embodiment enables log analysis to have high efficiency and high expansibility, is very suitable for docking new service scenes and products, can be quickly adapted to different log formats, and is convenient and fast to use.
Acquiring a log file to be analyzed, specifically:
and acquiring the path addresses of all log files, filtering the log files through the file names, and screening out the log files to be analyzed.
Preferably, before the log file is divided into a plurality of subfiles and each subfile is read and loaded, the method further includes:
and judging whether the log file is a compressed file, if so, decompressing the log file, and then moving the decompressed log file to an analysis temporary directory, otherwise, directly moving the log file to the analysis temporary directory.
After the log file is obtained, the log file is subjected to some preprocessing, i.e., decompression, file movement, and the like.
Preferably, the log file is divided into a plurality of subfiles, and each subfile is read and loaded, specifically:
judging whether the total line number of the log file is larger than a set line number, if so, cutting the log file into sub files with a plurality of set line numbers, otherwise, not cutting;
acquiring a path address of each subfile;
and reading each subfile in parallel or in series based on the path address, and loading the subfiles to a memory buffer area.
In this embodiment, first, whether the log file is greater than 50 ten thousand rows is determined, and if so, the log file is divided into a plurality of subfiles having 50 ten thousand rows; and then acquiring the path address of the cut subfile, and reading and loading the subfile in parallel or in series.
Specifically, as shown in fig. 3, the specific process of the method is as follows: s101, acquiring and storing a regular analysis object set from a risk component; s102, acquiring a log file to be analyzed; s201, before cutting a log file, judging whether the log is a compressed file, if so, turning to a step S202, otherwise, turning to a step S203; s202, decompressing the compressed file; s203, moving the log file to a temporary directory; s204, judging whether the log file is larger than the set line number, if so, turning to the step S205, otherwise, turning to the step S206; s205, cutting the log file; s206, loading and reading a log file; s3, the loaded subfiles are placed into a queue and distributed to each processing unit; s4, each processing unit analyzes the subfiles; s501, forming an sgn file based on the analysis result; and S502, warehousing the sgn file.
Preferably, the set number of lines and the size of the memory buffer are set according to configuration parameters of the distributed system.
The size of the subfile to be cut (i.e., the set number of rows) and the size of the memory buffer to load the subfile may be set according to the distributed system configuration to achieve the optimal performance configuration. Specifically, the higher the configuration of the distributed system is, the higher the calculation processing capability thereof is, so that the larger the set number of lines may be set, and the larger the memory buffer may be set correspondingly.
Preferably, the distributing the read loaded subfiles to a plurality of processing units in parallel further comprises:
and each processing unit distributes the log data in the corresponding sub-file to the peer processing unit or the next processing unit.
Specifically, as shown in fig. 4, in the distributed system, data (data) may be exchanged between the processing units to form a data stream (data stream), so that, when log analysis is performed by each processing unit, if a processing time is too long or a log file is unevenly distributed, an unprocessed sub-file may be sent to a processing unit of the same level or a processing unit of a next level for processing, thereby further improving log analysis efficiency, where a thin solid arrow in fig. 4 indicates a data stream between processing units of the same level, and a thick solid arrow indicates a data stream between processing units of the upper level and the lower level.
Preferably, each processing unit matches the regular parsing object of the corresponding subfile in the regular parsing object set, and performs parsing on the corresponding subfile, specifically:
each processing unit preliminarily analyzes the corresponding subfile, acquires a log header ip, and matches a corresponding regular analysis object in the regular analysis object set through the log header ip;
each processing unit carries out analysis field matching on the corresponding subfile according to the regular analysis object obtained by matching, and if the matching is successful, analysis field data of the corresponding subfile are extracted;
and combining the analysis field data obtained by matching each processing unit to form an analysis result.
Specifically, as shown in fig. 5, in the embodiment, when performing regular analysis object matching: s401, firstly, acquiring a subfile and a log header ip thereof; s402, acquiring a log source of the subfile through the header ip, further matching a regular object set corresponding to the log header ip of the subfile in the regular analysis object set, and turning to the step S404 if the matching is successful, or turning to the step S403 if the matching is not successful; s403, discarding the current subfile and recording an abnormal log; s404, circularly matching the sub-file with the regular objects in the corresponding regular analysis object set, if the matching is successful, turning to the step S405, otherwise, turning to the step S403; s405, extracting the regular objects successfully matched, and returning the matched analysis field data; and S406, after the current subfile is analyzed, transferring to the next subfile for analysis until all subfiles of the log file are analyzed.
Regular expression (regular expression) describes a pattern (pattern) for matching a character string, which can be used to check whether a string contains a certain substring, to replace the matching substring, or to extract a substring that meets a certain condition from a certain string, etc. The regular expression has the functions of successfully matching and capturing corresponding matching data (namely field data analysis); if the regular expression in the regular parsing object is successfully matched with the log file, predefined parsing fields in the log file are extracted.
The embodiment is realized by the log header ip matching mode, and the matching efficiency of the ip matching mode is high.
In this embodiment, the parsing process of the processing unit includes: adopting regular data in the subfile, carrying out regular verification on the regular data based on a regular expression, carrying out subscript matching on the regular data passing the verification after the regular verification passes, and carrying out type processing and/or internal and external network distinguishing on the regular data passing the verification and subscript matching; and generating an analysis result set according to the regular data processed by the data type and/or distinguished by the internal network and the external network.
Specifically, the specific step of using the regular data in the subfile is as follows: and collecting regular data of the subfiles from the regular data of the globally existing serialized logs by using a header ip. The type processing of the regular data specifically comprises: and converting the ip of the subfile into the long type and/or corresponding the protocol number of the subfile to the corresponding protocol type.
It should be understood that the parsing method provided by the present embodiment is applicable to log files generated by various different devices/systems.
Example 2
Embodiment 2 of the present invention provides a log file fast parsing apparatus, which includes a processor and a memory, where the memory stores a computer program, and when the computer program is executed by the processor, the log file fast parsing apparatus implements the log file fast parsing method provided in embodiment 1.
Specifically, as shown in fig. 6, the processor in this embodiment is divided into several functional modules, namely, a file acquisition unit 101, a file cutting unit 102, a file reading unit 103, a data distribution unit 104, a plurality of processing units 105, and a warehousing writing unit 106. The file obtaining unit 101 is configured to obtain path addresses of all log files, filter the path addresses by file names, and find out all log files to be analyzed; the file cutting unit 102 cuts a single oversized log file into a plurality of subfiles; the file reading unit 103 acquires the path address of the cut subfile, and reads and loads the subfile in parallel or in series; the data distribution unit 104 distributes the data loaded by the file reading unit 103 to the following processing unit 105 in parallel. The processing and analyzing unit 105 outputs a result set to form an sgn suffix file to be put in storage; the storage writing unit 106 performs storage writing operation on the result set output by the processing and analyzing unit 105, and writes the analysis result into the persistent data storage 2 for use.
The fast log file analyzing device provided by the embodiment of the invention is used for realizing the fast log file analyzing method, so that the technical effect of the fast log file analyzing method is achieved, and the fast log file analyzing device is also achieved, and is not repeated herein.
Example 3
Embodiment 3 of the present invention provides a computer storage medium having a computer program stored thereon, where the computer program, when executed by a processor, implements the log file fast parsing method provided in embodiment 1.
The computer storage medium provided by the embodiment of the invention is used for realizing the rapid analysis method of the log file, so that the computer storage medium has the technical effect of the rapid analysis method of the log file, and the description is omitted here.
The above description is only for the preferred embodiment of the present invention, but the scope of the present invention is not limited thereto, and any changes or substitutions that can be easily conceived by those skilled in the art within the technical scope of the present invention are included in the scope of the present invention.
Claims (10)
1. A method for rapidly analyzing log files is applied to a distributed system and comprises the following steps:
acquiring and storing a regular analysis object set, and acquiring a log file to be analyzed;
cutting the log file into a plurality of subfiles, and reading and loading each subfile;
distributing the read and loaded subfiles to a plurality of processing units in parallel;
each processing unit respectively matches the regular analysis object of the corresponding subfile in the regular analysis object set and analyzes the corresponding subfile;
and performing warehousing operation on the analysis results of each processing unit.
2. The method for rapidly parsing a log file according to claim 1, further comprising:
and after the analysis processing of all the subfiles of one log file is finished, setting time for dormancy, and then analyzing the next log file.
3. The method for rapidly parsing a log file according to claim 1, wherein a regular parsing object set is obtained and stored, specifically:
and setting a special storage space, receiving the regular analysis object set sent by the java component through an activeMq message queue, and storing the regular analysis object set to the special storage space.
4. The method for rapidly parsing a log file according to claim 1, wherein before the log file is divided into a plurality of subfiles and each subfile is read and loaded, the method further comprises:
and judging whether the log file is a compressed file, if so, decompressing the log file, and then moving the decompressed log file to an analysis temporary directory, otherwise, directly moving the log file to the analysis temporary directory.
5. The method for rapidly parsing a log file according to claim 1, wherein the log file is divided into a plurality of subfiles, and each subfile is read and loaded, specifically:
judging whether the total line number of the log file is larger than a set line number, if so, cutting the log file into sub files with a plurality of set line numbers, otherwise, not cutting;
acquiring a path address of each subfile;
and reading each subfile in parallel or in series based on the path address, and loading the subfiles to a memory buffer area.
6. The method according to claim 5, wherein the set number of lines and the size of the memory buffer are set according to configuration parameters of a distributed system.
7. The method for rapidly parsing log files according to claim 1, wherein the read and loaded subfiles are distributed to a plurality of processing units in parallel, further comprising:
and each processing unit distributes the log data in the corresponding sub-file to the peer processing unit or the next processing unit.
8. The method according to claim 1, wherein each processing unit matches a regular parsing object of a corresponding subfile in the regular parsing object set, and performs parsing on the corresponding subfile, specifically:
each processing unit preliminarily analyzes the corresponding subfile, acquires a log header ip, and matches a corresponding regular analysis object in the regular analysis object set through the log header ip;
each processing unit carries out analysis field matching on the corresponding subfile according to the regular analysis object obtained by matching, and if the matching is successful, analysis field data of the corresponding subfile are extracted;
and combining the analysis field data obtained by matching each processing unit to form an analysis result.
9. A fast log file parsing device, comprising a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the fast log file parsing device implements the fast log file parsing method according to any one of claims 1 to 8.
10. A computer storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the log file fast parsing method according to any one of claims 1 to 8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011626206.7A CN112612767A (en) | 2020-12-30 | 2020-12-30 | Log file rapid analysis method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011626206.7A CN112612767A (en) | 2020-12-30 | 2020-12-30 | Log file rapid analysis method and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112612767A true CN112612767A (en) | 2021-04-06 |
Family
ID=75253033
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011626206.7A Pending CN112612767A (en) | 2020-12-30 | 2020-12-30 | Log file rapid analysis method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112612767A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112988397A (en) * | 2021-04-20 | 2021-06-18 | 浙江贵仁信息科技股份有限公司 | Rapid analysis method and conversion server for water conservancy model file |
-
2020
- 2020-12-30 CN CN202011626206.7A patent/CN112612767A/en active Pending
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112988397A (en) * | 2021-04-20 | 2021-06-18 | 浙江贵仁信息科技股份有限公司 | Rapid analysis method and conversion server for water conservancy model file |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN104331435B (en) | A kind of efficient mass data abstracting method of low influence based on Hadoop big data platforms | |
| CN101546342B (en) | Method and system for implementing search service | |
| CN112468370A (en) | High-speed network message monitoring and analyzing method and system supporting custom rules | |
| CN107590054B (en) | Ship server log monitoring system | |
| CN109800259B (en) | Data acquisition method and device and terminal equipment | |
| CN110690984A (en) | Spark-based big data weblog acquisition, analysis and early warning method and system | |
| US20150074115A1 (en) | Distributed storage of data | |
| CN111125013B (en) | Data warehousing method, device, equipment and medium | |
| CN108228322B (en) | Distributed link tracking and analyzing method, server and global scheduler | |
| CN1859505B (en) | Bill checking system and method | |
| CN105677615A (en) | Distributed machine learning method based on weka interface | |
| CN111726252A (en) | Abnormal data reporting method and tool for industrial equipment | |
| CN114416670A (en) | Index creating method and device suitable for network disk document, network disk and storage medium | |
| CN112612767A (en) | Log file rapid analysis method and device | |
| CN110008701B (en) | Static detection rule extraction method and detection method based on ELF file characteristics | |
| CN109194755B (en) | MQ-based mobile device data processing method and system | |
| CN109033295B (en) | Method and device for merging super-large data sets | |
| CN106980618A (en) | File memory method and system based on MongoDB distributed type assemblies frameworks | |
| CN111130882A (en) | Monitoring system and method of network equipment | |
| CN111078975B (en) | Multi-node incremental data acquisition system and acquisition method | |
| CN115664992A (en) | Network operation data processing method and device, electronic equipment and medium | |
| US20150081649A1 (en) | In-line deduplication for a network and/or storage platform | |
| CN113872814A (en) | Information processing method, device and system for content distribution network | |
| CN113360576A (en) | Power grid mass data real-time processing method and device based on Flink Streaming | |
| CN113590447A (en) | Buried point processing method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |