CN111061628A - Data analysis method, system, device, computer equipment and storage medium - Google Patents
Data analysis method, system, device, computer equipment and storage medium Download PDFInfo
- Publication number
- CN111061628A CN111061628A CN201911147117.1A CN201911147117A CN111061628A CN 111061628 A CN111061628 A CN 111061628A CN 201911147117 A CN201911147117 A CN 201911147117A CN 111061628 A CN111061628 A CN 111061628A
- Authority
- CN
- China
- Prior art keywords
- abnormal data
- analysis
- data
- client
- server
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/36—Preventing errors by testing or debugging software
- G06F11/3668—Software testing
- G06F11/3672—Test management
- G06F11/3692—Test management for test results analysis
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/1805—Append-only file systems, e.g. using logs or journals to store data
- G06F16/1815—Journaling file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2291—User-Defined Types; Storage management thereof
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02P—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS
- Y02P90/00—Enabling technologies with a potential contribution to greenhouse gas [GHG] emissions mitigation
- Y02P90/30—Computing systems specially adapted for manufacturing
Abstract
The application relates to a data analysis method, a system, a device, a computer device and a storage medium. The method comprises the following steps: the client acquires a log file corresponding to software operation, integrates abnormal data in the log file to obtain integrated abnormal data, and sends the integrated abnormal data to the server; the comprehensive analysis at least comprises problem feature analysis, vulnerability and non-functional problem analysis and performance analysis of abnormal data. In the method, the whole process of comprehensively acquiring all error and abnormal information of a software system and carrying out centralized cleaning, supplementing and analyzing on data is realized, and the problems of inconvenience, redundancy, easiness in omission, low efficiency in partial problem solving and the like of manual query are solved; and abnormal data are analyzed in a centralized manner from different dimensions, so that the analysis result of the abnormal data is more accurate, the directivity is more obvious, and the analysis standardization requirement is met.
Description
Technical Field
The present application relates to the field of software technologies, and in particular, to a data analysis method, system, apparatus, computer device, and storage medium.
Background
The error and abnormal information in the software running process usually reflect that the software runs abnormally or has errors, if all the errors and the errors can be eliminated, the software running result is in accordance with the expectation of a user, the software stability is good, if the problems can be quickly found and timely solved at the initial stage of the error occurrence or under the condition that the error amount is not large, the software quality can be effectively improved.
Most server applications, system software, and Android, IOS applications now provide a relatively sophisticated mechanism to export this information into a file called a log file. However, software errors and abnormal data have great randomness, the content and format of the data are often different from those of other data, and meanwhile, the proportion of the data in a file is often not large, so that the data are difficult to accurately collect in a uniform mode, and the data are analyzed subsequently. At present, the most common technology is to search information related to problems from log files through a series of commands of Linux, and to implement complex query of log contents to a certain extent by combining with a pipeline, and the query result is generally directly output to a display screen or output to a file, and then is manually analyzed after finding out error or abnormal information.
However, the above method for manually analyzing the abnormal information in the log is inefficient, and cannot meet the requirement of analyzing and standardizing the abnormal data.
Disclosure of Invention
In view of the above, it is necessary to provide a data analysis method, system, apparatus, computer device and storage medium for solving the above technical problems.
In a first aspect, the present application provides a data analysis method, including:
the client acquires a log file corresponding to software operation;
the client side integrates abnormal data in the log file to obtain integrated abnormal data; the integration processing comprises merging the abnormal data and adding the effective field of the abnormal data; the valid field at least comprises a product to which the abnormal data belongs, an item to which the abnormal data belongs, a client ID and a server address;
the client side sends the abnormal data after the integration processing to the server side; the abnormal data is used for indicating the server to comprehensively analyze the abnormal data after the integration processing to obtain an analysis result; the comprehensive analysis at least comprises problem feature analysis, vulnerability and non-functional problem analysis and performance analysis of abnormal data.
In one embodiment, the above-mentioned client performs integration processing on the abnormal data in the log file to obtain the abnormal data after the integration processing, and the method includes:
the client acquires abnormal data of the log file according to the data format identifier in the log file;
and the client side combines all abnormal data in the abnormal data of the log file into a complete abnormal data respectively, and adds the effective field of the abnormal data to obtain the abnormal data after the integration processing.
In one embodiment, the sending, by the client, the abnormal data after the integration processing to the server includes:
the client sends the abnormal data after the integration processing to the server through the transfer terminal; the transfer terminal is used for analyzing and packaging the abnormal data after the integration processing, and sending the abnormal data after the analysis and packaging processing to the server terminal; wherein, different client types correspond to different analysis and encapsulation processing modes.
In one embodiment, the parsing and packaging process includes:
analyzing the abnormal data after the integration processing to obtain a target field, and packaging the target field; the target field at least comprises the occurrence time of the abnormal data, the belonged product, the belonged item, the belonged class, the belonged method, the belonged row, the abnormal type and the address of the server.
In one embodiment, the method further comprises:
the client verifies the validity of the function configuration of the log file;
and if the function configuration is legal, establishing network connection with the transfer terminal, and sending the abnormal data after the integration processing to the transfer terminal.
In one embodiment, after the client sends the abnormal data after the integration processing to the server, the method further includes:
the client stores the processing information of the log file in a database; the processing information of the log file at least comprises a file path, a file name, a file type, the line number of the last line in the log and a timestamp corresponding to the last line.
In a second aspect, the present application provides a data analysis method, comprising:
the server receives the abnormal data after the integration processing sent by the client, and the abnormal data after the integration processing is data obtained by integrating and processing the abnormal data in the log file corresponding to the software operation by the client; the integration processing comprises merging the abnormal data and adding the effective field of the abnormal data; the valid field at least comprises a product to which the abnormal data belongs, an item to which the abnormal data belongs, a client ID and a server address;
the server side comprehensively analyzes the abnormal data after the integration processing to obtain an analysis result; the comprehensive analysis at least comprises problem feature analysis, vulnerability and non-functional problem analysis and performance analysis of abnormal data.
In one embodiment, the performing, by the server, comprehensive analysis on the abnormal data after the integration processing includes:
the server side carries out validity check on the abnormal data after the integration processing;
and if the abnormal data after the integration processing is effective, executing a step of comprehensively analyzing the abnormal data after the integration processing.
In one embodiment, after obtaining the analysis result, the method further comprises:
and the server stores and displays the analysis result.
In a third aspect, the present application provides a data analysis system, comprising:
a client configured to perform the method provided in any embodiment of the first aspect;
and the server is used for executing the method provided by any embodiment of the second aspect.
In a fourth aspect, the present application provides a data analysis apparatus, comprising:
the acquisition module is used for acquiring a log file corresponding to software operation by a client;
the processing module is used for integrating the abnormal data in the log file by the client to obtain the integrated abnormal data; the integration processing comprises merging the abnormal data and adding the effective field of the abnormal data; the valid field at least comprises a product to which the abnormal data belongs, an item to which the abnormal data belongs, a client ID and a server address;
the sending module is used for sending the abnormal data after the integration processing to the server side by the client side; the abnormal data is used for indicating the server to comprehensively analyze the abnormal data after the integration processing to obtain an analysis result; the comprehensive analysis at least comprises problem feature analysis, vulnerability and non-functional problem analysis and performance analysis of abnormal data.
In a fifth aspect, the present application provides a data analysis apparatus, comprising:
the receiving module is used for receiving the abnormal data after the integration processing sent by the client side by the server side, and the abnormal data after the integration processing is data obtained by integrating and processing the abnormal data in the log file corresponding to the software operation by the client side; the integration processing comprises merging the abnormal data and adding the effective field of the abnormal data; the valid field at least comprises a product to which the abnormal data belongs, an item to which the abnormal data belongs, a client ID and a server address;
the analysis module is used for comprehensively analyzing the abnormal data after the integration processing by the server side to obtain an analysis result; the comprehensive analysis at least comprises problem feature analysis, vulnerability and non-functional problem analysis and performance analysis of abnormal data.
In a sixth aspect, the present application provides a computer device, including a memory and a processor, where the memory stores a computer program, and the processor implements the data analysis method provided in any one of the embodiments of the first and second aspects when executing the computer program.
In a seventh aspect, the present application provides a computer-readable storage medium, on which a computer program is stored, where the computer program, when executed by a processor, implements the data analysis method provided in any one of the embodiments of the first and second aspects.
According to the data analysis method, the data analysis system, the data analysis device, the computer equipment and the storage medium, the log file corresponding to software operation is obtained through the client, the abnormal data in the log file is integrated to obtain the integrated abnormal data, and the integrated abnormal data is sent to the server. The abnormal data are used for indicating the server to comprehensively analyze the abnormal data after the integration processing to obtain an analysis result; the comprehensive analysis at least comprises problem feature analysis, vulnerability and non-functional problem analysis and performance analysis of abnormal data. According to the method, the client automatically acquires abnormal data in the log file for integration processing, and sends the integrated abnormal data to the server to indicate the server to comprehensively analyze the abnormal data, so that the whole process can comprehensively acquire all error and abnormal information of a software system and perform centralized cleaning, supplementing and analyzing on the data, and the problems of inconvenience, redundancy, easiness in omission, low efficiency in partial problem solving and the like of manual query are solved; and abnormal data are analyzed in a centralized manner from different dimensions, so that the analysis result of the abnormal data is more accurate, the directivity is more obvious, and the analysis standardization requirement is met.
Drawings
FIG. 1 is a diagram of an exemplary data analysis application;
FIG. 2 is a schematic flow chart diagram of a data analysis method in one embodiment;
FIG. 3 is a schematic flow chart diagram of a data analysis method in one embodiment;
FIG. 4 is a system diagram of a data analysis method in accordance with one embodiment;
FIG. 5 is a schematic flow chart diagram illustrating a data analysis method in one embodiment;
FIG. 6 is a schematic flow chart diagram of a data analysis method in another embodiment;
FIG. 7 is a schematic flow chart diagram illustrating a data analysis method in one embodiment;
FIG. 7a is a schematic diagram of an analysis result presentation interface of the data analysis method in one embodiment;
FIG. 7b is a schematic diagram of an analysis result presentation interface of the data analysis method in one embodiment;
FIG. 8 is a block diagram of the structure of a data analysis system in one embodiment;
FIG. 9 is a block diagram of the structure of a data analysis system in one embodiment;
FIG. 10 is a block diagram showing the structure of a data analysis apparatus according to an embodiment;
FIG. 11 is a block diagram showing the structure of a data analysis apparatus according to an embodiment;
FIG. 12 is a block diagram showing the structure of a data analysis device according to an embodiment;
FIG. 13 is a block diagram showing the structure of a data analysis apparatus according to an embodiment;
FIG. 14 is a block diagram showing the structure of a data analysis apparatus according to an embodiment;
FIG. 15 is a block diagram showing the construction of a data analysis apparatus according to another embodiment;
FIG. 16 is a block diagram showing the structure of a data analysis apparatus according to an embodiment;
FIG. 17 is a block diagram showing the structure of a data analysis apparatus according to an embodiment;
FIG. 18 is a diagram illustrating an internal structure of a computer device according to an embodiment.
Detailed Description
In order to make the objects, technical solutions and advantages of the present application more apparent, the present application is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application.
The data analysis method provided by the application can be applied to the structure of the data analysis system shown in FIG. 1. The data analysis system may include at least one client 101 and at least one server 102, where the client 101 and the server 102 communicate via a network. The client 101 may be, but not limited to, various personal computers, notebook computers, smart phones, tablet computers, and portable wearable devices, and the server 102 may be implemented by an independent server or a server cluster formed by a plurality of servers.
The following describes in detail the technical solutions of the present application and how the technical solutions of the present application solve the above technical problems by embodiments and with reference to the drawings. The following several specific embodiments may be combined with each other, and details of the same or similar concepts or processes may not be repeated in some embodiments. It should be noted that, in the data analysis method provided in the embodiments of fig. 2 to fig. 5 of the present application, the execution subject is a client; 6-7, the execution subject is the server side; in the data analysis method provided in the embodiments of fig. 2 to fig. 7, the execution subject may also be a data analysis apparatus, and the data analysis apparatus may become part or all of the client and/or the server by software, hardware, or a combination of software and hardware.
In the following method embodiments, an embodiment in which the execution subject is a client will be described first.
In an embodiment, as shown in fig. 2, a data analysis method is provided, which is described by taking an example that the method is applied to a client in fig. 1, where the embodiment relates to a specific process that the client acquires a log file corresponding to software operation, performs integration processing on abnormal data in the log file to obtain integrated processed abnormal data, and sends the integrated processed abnormal data to a server, so that the server performs comprehensive analysis on the integrated processed abnormal data to obtain an analysis result, and the embodiment includes the following steps:
s201, the client acquires a log file corresponding to software operation.
The log file refers to a record file for recording software operation events; the log file comprises normal software operation data, abnormal data generated during software operation, program error information generated during software operation and the like.
In this embodiment, the client acquires a complete log file, wherein after acquiring the log file, the client can monitor whether new data is generated in the log file in real time, and can also check whether new data is generated in the log file in a timing manner; for example, the client may check whether the log file is updated at regular time, set the time interval to be 100ms, and the client may scan whether the log file is updated every 100 ms; the method can also be used for scanning the log file again according to the multiple of the set time interval after checking that the file is not updated for several times in an accumulation manner until new data of the log file is scanned. In the process, the log file information specifically checked by the client includes the update time of the file in the log file, the file size, the file line number, and the content of the last line of the file, and the obtained contents are compared with the content obtained last time, if the contents are not consistent, it is indicated that the log file generates new data, and the new data in the log file is obtained, which is not limited in this embodiment.
S202, integrating abnormal data in the log file by the client to obtain the integrated abnormal data; the integration processing comprises merging the abnormal data and adding the effective field of the abnormal data; the valid field at least comprises a product to which the abnormal data belongs, an item to which the abnormal data belongs, a client ID and a server address.
The abnormal data comprises software running abnormity and program error information; the exception refers to an exception event occurring during the running process of the program, and is usually caused by an external problem (such as a hardware error and an input error), and the program error refers to phenomena such as abnormal function, dead halt, data loss, abnormal interruption and the like caused by an error of the program itself or incapability of normally accessing necessary resources during the running process of the software. Both exceptions and program errors may be embodied in the form of data in a log file; valid fields refer to data information needed to analyze anomalous data from different dimensions.
In this embodiment, the client obtains the abnormal data from the new data of the log file, and the obtaining basis may be a preset abnormal data format type or a character identifier of the abnormal data; and after the client side acquires the abnormal data, integrating the abnormal data. Generally, an abnormal data is composed of multiple lines of data, and after acquiring the multiple lines of data, a client merges the multiple lines of data into one line, and one line of data represents an abnormal data. Similarly, the merging base may be a character identifier of the abnormal data or a format type of the abnormal data, and particularly, if the abnormal data does not include the above two merging bases, the abnormal data is merged with the normal operation log in the previous row, and the time stamp of the normal log is used as the approximate time stamp of the abnormal data. Generally, the abnormal data may not include information data required by the analysis from different dimensions in the present scheme, so according to the analysis requirement, an effective field set is set, which at least includes a product to which the abnormal data belongs, an item to which the abnormal data belongs, a client ID, and a server address, the client adds the effective field to the merged abnormal data, and the client may add to the front section of the abnormal data or add to the back section of the abnormal data, which is not limited in this embodiment.
S203, the client sends the abnormal data after the integration processing to the server; the abnormal data is used for indicating the server to comprehensively analyze the abnormal data after the integration processing to obtain an analysis result; the comprehensive analysis at least comprises problem feature analysis, vulnerability and non-functional problem analysis and performance analysis of abnormal data.
The abnormal data after the integration processing is abnormal data comprising effective fields; problem characteristic analysis of abnormal data refers to analyzing the characteristics of specific program errors and abnormal data, including items to which the program errors and abnormal data belong, code modules where the program errors and abnormal data belong, error code, content contained in error and abnormal information, error amount and error rate threshold values, abnormality in how long time, number of errors and increment thereof, and Boolean logic between abnormal data; vulnerability and non-functional problem analysis refers to regularly checking the variation trend of error and abnormal data within a certain time interval; the performance analysis refers to the centralized analysis of occurrence trend of abnormal data, abnormal trend according to products, projects and servers, abnormal classification, distribution of the abnormal on each project and detailed information of the abnormal from four dimensions of the whole, the projects, the servers and the abnormal.
In this embodiment, the client sends the integrated abnormal data to the server through different ways, the client can directly send the integrated abnormal data to the corresponding server according to the server address in the valid field, and can also send the integrated abnormal data to the relay, and the relay further processes the integrated abnormal data and sends the processed abnormal data to the corresponding server. The abnormal data is used for indicating the server side to analyze the abnormal data from different dimensions.
Optionally, after the client sends the processed abnormal data to the server, the client further stores the processing information of the log file in a database; the processing information of the log file at least comprises a file path, a file name, a file type, the line number of the last line in the log and a timestamp corresponding to the last line.
In this embodiment, after the client checks that the last abnormal data processing is completed, the client records the relevant information of the processed abnormal data in the log file, where the relevant information is the complete path, the file name, the file type, the line number of the last line, and the timestamp corresponding to the last line of the log file. The client may store the relevant information in the database, or store the relevant information in a local file, and when the client reads the log file next time, the client reads data from the next row according to the relevant information, and if the relevant information cannot be found, the client reads data from the first row of the log file, which is not limited in this embodiment.
In the data analysis method, the client acquires the log file corresponding to the software operation, integrates the abnormal data in the log file to obtain the integrated abnormal data, and sends the integrated abnormal data to the server. The abnormal data are used for indicating the server to comprehensively analyze the abnormal data after the integration processing to obtain an analysis result; the comprehensive analysis at least comprises problem feature analysis, vulnerability and non-functional problem analysis and performance analysis of abnormal data. In the embodiment, the client automatically acquires the abnormal data in the log file for integration processing, and sends the integrated abnormal data to the server to indicate the server to comprehensively analyze the abnormal data, so that the whole process realizes the comprehensive acquisition of all error and abnormal information of a software system, and the centralized cleaning, supplement and analysis of the data are realized, and the problems of inconvenience, redundancy, easiness in omission, low efficiency in partial problem solution and the like of manual query are solved; and abnormal data are analyzed in a centralized manner from different dimensions, so that the analysis result of the abnormal data is more accurate, the directivity is more obvious, and the analysis standardization requirement is met.
When the client side obtains the log file, the client side needs to perform integration processing on the abnormal data in the log file, one implementation manner of which may be as shown in fig. 3, in an embodiment, the client side performs integration processing on the abnormal data in the log file to obtain the integrated abnormal data, and the embodiment includes:
s301, the client acquires abnormal data of the log file according to the data format identification in the log file.
The log file comprises normal operation data and abnormal data, and the data format identifier is a basis for distinguishing the operation data from the abnormal data.
In this embodiment, the normal operation data and the abnormal data in the log file have different data format identifiers, for example, the abnormal data generally consists of a timestamp, a packet name where the abnormal data is located, and data beginning with "at", according to the data format identifier unique to the abnormal data, the client reads the abnormal data from the log file, the different abnormal data have different data format identifiers, and the client can obtain different abnormal data according to the preset data format identifiers of different types, which is not limited in this embodiment.
S302, the client side combines all abnormal data in the abnormal data of the log file into a complete abnormal data, and adds an effective field of the abnormal data to obtain the abnormal data after integration processing.
Wherein the valid field refers to data information required for analyzing the abnormal data from different dimensions. The method at least comprises a product to which the abnormal data belongs, an item to which the abnormal data belongs, a client ID and a server address.
In this embodiment, generally, the abnormal data in the log file exists in a form of multiple rows of data, each row of data serves as a character string, and the client may merge multiple rows of data of one abnormal data into one row of data according to the data format identifier of the abnormal data, so that one abnormal data is presented in a form of one character string. After merging the abnormal data, adding necessary effective fields to the abnormal data so that the server side performs comprehensive analysis according to the abnormal data and the effective fields.
In this embodiment, because the abnormal data in the original log file exists in the form of multiple rows of data, if the client sends an abnormal data to the server, each row of data is sent as a character string, which may cause the incompleteness of the abnormal data, so that multiple rows of data of one abnormal data are combined and an effective field is added for sending, thereby ensuring the completeness of the abnormal data and providing a reliable basis for the server to comprehensively analyze the abnormal data.
After processing the abnormal data, the client sends the processed abnormal data to the server, and the sending method may be directly sent to the server, or may be further processed by the relay and then forwarded to the server, as shown in fig. 4, which includes a data analysis system architecture diagram of the relay, in an embodiment, one implementation manner includes:
the client sends the abnormal data after the integration processing to the server through the transfer terminal; the transfer terminal is used for analyzing and packaging the abnormal data after the integration processing, and sending the abnormal data after the analysis and packaging processing to the server terminal; wherein, different client types correspond to different analysis and encapsulation processing modes.
The analysis processing refers to analyzing the abnormal data into a series of fields and converting the fields into corresponding data types, and the encapsulation processing refers to encapsulating each field of data into a group of complete abnormal data.
In this embodiment, the client sends the abnormal data after the integration processing to the transfer terminal, and the transfer terminal further processes the abnormal data, where the processing includes parsing and packaging, and after the processing is completed, the parsed and packaged abnormal data is sent to the server. When the transit terminal accesses each client, the type of the client may be configured, and when data is subsequently analyzed and processed, different data analysis processing logics may be respectively adopted according to the type, the client ID, and the address of the server where the client is located, which is not limited in this embodiment.
Preferably, the transit terminal may further perform validity check on the abnormal data after the integration processing, for example, filter out data that does not belong to the abnormal feature, and do not perform processing. The Exception features include, for example, Java exceptions, which tend to contain exceptions and have multiple lines merged into one line; for program error data, a regular expression can be adopted for matching by combining the characteristics of specific items.
Optionally, the analyzing and encapsulating process may include analyzing the abnormal data after the integrating process, obtaining a target field, and encapsulating the target field; the target field at least comprises the occurrence time of the abnormal data, the belonged product, the belonged item, the belonged class, the belonged method, the belonged row, the abnormal type and the address of the server.
Specifically, after the client sends the integrated abnormal data to the relay terminal, the relay terminal analyzes the abnormal data according to a target field, that is, a target field in the abnormal data is analyzed, the target field can be an effective field itself or a redefined field set, and the specific target field can include occurrence time, error information, a code, a product, a project and a server address; the value of the exception data and specific data information can also be the exception message, the class where the exception is located, the method, which line of the code triggering the exception is, and the exception stack. After the analysis is completed, the transfer terminal encapsulates the field data, sends the analyzed and encapsulated abnormal data to the server, and simultaneously, the transfer terminal can simultaneously send the analyzed and encapsulated abnormal data records to the server for storage.
Further, before the client forwards the abnormal data through the transit terminal, the client needs to establish a network connection with the transit terminal, and in an embodiment, as shown in fig. 5, the process of establishing the network connection may include:
s501, the client verifies the validity of the function configuration of the log file.
The validity of the configuration of the verification function refers to verifying the validity of a configuration block of the log file, wherein the configuration block comprises a path and a naming rule of the file of the log file, an error and abnormal information judgment rule, a control configuration block starting switch, a transit service domain name, a port or an address and the like.
In this embodiment, the client verifies the validity of the function configuration block in the log file, that is, verifies whether the path and name of the file in the log file conform to the preset naming rule, verifies whether the judgment rule of the error and abnormal information in the log file is reasonable, verifies whether the setting of the activation switch of the control configuration block is completed, and verifies whether the domain name, the port or the address of the transit service is empty, which is not limited in this embodiment.
And S502, if the function configuration is legal, establishing network connection with the transfer terminal, and sending the abnormal data after the integration processing to the transfer terminal.
In this embodiment, after verifying that the function configuration of the log file is legal, the client establishes a connection with the relay terminal according to the domain name, the port, or the address of the relay service, and the network connection may be performed through various wireless protocols. And after the connection is confirmed to be successful, the client sends the processed abnormal data to the transfer terminal and indicates the transfer terminal to execute the steps of analyzing and packaging.
It should be noted that, if the implementation manner includes the relay end, the relay end executes the method steps in the foregoing embodiment, and if the implementation manner does not include the relay end, the method steps in the foregoing embodiment may be implemented at the client or at the server, which is not limited in this embodiment.
In this embodiment, the client can send the integrated processed abnormal data to the server through the transit terminal, and the transit terminal plays a role in bearing, so that the sent abnormal data of the client is further processed, the data processing pressure of the server is reduced, and the interaction efficiency of the client and the server is improved.
The application of the data analysis method is explained in detail by taking the client as an example, and in the following method embodiment, an embodiment in which the execution subject is the server is explained.
In an embodiment, as shown in fig. 6, a data analysis method is provided, which is described by taking the application of the method to the server 102 in fig. 1 as an example, and the embodiment relates to a specific process in which the server receives abnormal data after integration processing sent by the client, and performs comprehensive analysis on the abnormal data after integration processing to obtain an analysis result, including the following steps:
s601, the server receives the abnormal data after integration processing sent by the client, and the abnormal data after integration processing is data obtained by integrating and processing the abnormal data in the log file corresponding to the software operation by the client; the integration processing comprises merging the abnormal data and adding the effective field of the abnormal data; the valid field at least comprises a product to which the abnormal data belongs, an item to which the abnormal data belongs, a client ID and a server address.
In this embodiment, the server may receive, through a wireless network of any one protocol, the abnormal data after the integration processing sent by the client, where the abnormal data includes an effective field for merging the abnormal data and adding the abnormal data, and the effective field may be used as a basis for the server to perform comprehensive analysis on the abnormal data.
S602, comprehensively analyzing the abnormal data after the integration processing by the server side to obtain an analysis result; the comprehensive analysis at least comprises problem feature analysis, vulnerability and non-functional problem analysis and performance analysis of abnormal data.
The problem characteristic analysis of the abnormal data refers to analyzing the characteristics of specific program errors and abnormal data, including items to which the program errors and abnormal data belong, content contained in code modules where the program errors and abnormal data exist, error amount and error rate threshold values, abnormality in a long time, the number of errors and increment thereof, and Boolean logic between the abnormal data; vulnerability and non-functional problem analysis refers to regularly checking the variation trend of error and abnormal data within a certain time interval; the performance analysis refers to the centralized analysis of occurrence trend of abnormal data, abnormal trend according to products, projects and servers, abnormal classification, distribution of the abnormal on each project and detailed information of the abnormal from four dimensions of the whole, the projects, the servers and the abnormal.
In this embodiment, the server performs comprehensive analysis on the received processed abnormal data, where the problem feature analysis specifically includes the discovery of the problem reflected by the error and abnormal data through a preset rule. The types of the preset rules include: a particular error, an exception in a time period, the number of error occurrences and their increments, an increment in an adjacent time period, a boolean logical combination between questions, etc., illustratively, if two exceptions occur at the same time, then an association is assumed between the two exceptions. Bug and non-functional problem analysis is specifically that a server side checks the change trend of error and abnormal data in a certain time interval at regular time during running, if a set condition is met, the system is determined to be not normal in running, and at the moment, error and abnormal data meeting a rule and problem data based on the abnormal and abnormal data with more occurrence times are generated, and further, the server side can output a detailed list of problems in the form of a test report. The performance analysis specifically includes that the server side performs centralized analysis on occurrence trends of the abnormalities, abnormal trends according to products, projects and servers, abnormal classification, distribution of the abnormalities on each project and detailed information of the abnormalities from four dimensions of the whole situation, the projects, the servers and the abnormalities. Preferably, after performing comprehensive analysis on each abnormal data, the server may automatically generate and record detailed information of the analysis result, wherein the detailed information includes the grade, the nature description, the suggestion and the like of the abnormality.
In this embodiment, after receiving the abnormal data after the integration processing sent by the client, the server performs comprehensive analysis on the abnormal data from the aspects of problem feature analysis, vulnerability and non-functional problem analysis, performance analysis and the like of the abnormal data according to analysis standards of different dimensions, replaces the subjectivity of the traditional manual analysis, increases the reliability of the abnormal data analysis, and meets the standardized requirements of the abnormal data analysis.
After receiving the abnormal data, the server performs comprehensive analysis on the abnormal data, as shown in fig. 7, in an embodiment, the performing comprehensive analysis on the abnormal data after the integration processing by the server includes:
and S701, the server checks the validity of the abnormal data after the integration processing.
In this embodiment, the validity check may be to verify whether the abnormal data after the integration processing is empty, or may be to verify whether the abnormal data after the integration processing conforms to a preset abnormal data rule. Illustratively, after receiving the abnormal data after the integration processing, the server checks whether the abnormal data is empty.
And S702, if the abnormal data after the integration processing is valid, executing a step of comprehensively analyzing the abnormal data after the integration processing.
In this embodiment, the server checks whether the abnormal data is empty, and if the abnormal data is not empty, it indicates that the integrated abnormal data is valid, and performs comprehensive analysis on the abnormal data.
In this embodiment, the server performs validity check on the received abnormal data after the integration processing, so that the validity of the abnormal data is ensured, the validity of the data processed by the server is also ensured, and the execution efficiency of the server is improved.
After the server performs comprehensive analysis on the abnormal data, an analysis result is obtained, and in one embodiment, the server may store and display the analysis result.
In this embodiment, after the server performs comprehensive analysis on the abnormal data, analysis results of different dimensions are obtained, and the analysis results are stored and displayed, where the analysis results may be stored in a local file or a database, and the database may be a non-relational database Not Only SQL. For example, as shown in fig. 7a and 7b, the server may visually and quantitatively present the analysis results of different dimensions in a display interface in a graphical form, so that a worker can clearly see the analysis results of abnormal data in the software running from different dimensions, for example, see information such as an abnormal occurrence trend, an occurrence frequency, an occurrence reason, an abnormal classification, and a coupling between abnormalities from the display interface, which is not limited in this embodiment.
In this embodiment, the server can store the analysis result of the abnormal data, and then display the analysis result from different dimensions by reading the stored abnormal data analysis result, so that the analysis result is more visual, and the requirement for looking up the historical analysis result at any time is met.
It should be understood that although the various steps in the flow charts of fig. 1-7 are shown in order as indicated by the arrows, the steps are not necessarily performed in order as indicated by the arrows. The steps are not performed in the exact order shown and described, and may be performed in other orders, unless explicitly stated otherwise. Moreover, at least some of the steps in fig. 1-7 may include multiple sub-steps or multiple stages that are not necessarily performed at the same time, but may be performed at different times, and the order of performance of the sub-steps or stages is not necessarily sequential, but may be performed in turn or alternating with other steps or at least some of the sub-steps or stages of other steps.
In one embodiment, in conjunction with the block diagram of the data analysis system shown in fig. 1, as shown in fig. 8, there is provided a data analysis system comprising: client 801 and server 802, wherein:
a client 801, configured to execute the method provided by any embodiment with the client as an execution subject; the client 801 may be a computer device, or may be a tool that occupies tens of Mb in memory and has a CPU occupancy rate controlled to about 2%, and the tool may be directly deployed on a computer device in a test environment or a production environment.
A server 802, configured to execute the method provided by any embodiment with the server as an execution subject; the server 802 may be an independent server or a server cluster composed of a plurality of servers.
According to the method provided by the foregoing embodiment, in an implementation manner, as shown in fig. 9, the data analysis system may further include a transit terminal 803;
the transit terminal 803 is configured to analyze and encapsulate the integrated abnormal data, and send the analyzed and encapsulated abnormal data to the server; different client types correspond to different analysis and packaging processing modes; the relay terminal may be a Remote Procedure Call (RPC) server, and the exception data sent by the client terminal is analyzed, encapsulated, and then sent to the server terminal by calling an HTTP interface. It can be understood that the transit terminal and the service terminal may be disposed on a terminal or may be disposed separately, which is not limited in this embodiment.
The implementation principle and technical effect of the data analysis system provided by the above embodiment are similar to those of the above method embodiment, and are not described herein again.
For specific limitations of the data analysis system, reference may be made to the above limitations of the data analysis method, which are not described herein again. The client 801 in the data analysis system may be, but is not limited to, various personal computers, notebook computers, smart phones, tablet computers, and portable wearable devices; the server 802 may be implemented by an independent server or a server cluster composed of a plurality of servers; the transit terminal 803 may also be implemented by a separate server or a server cluster composed of a plurality of servers.
In one embodiment, as shown in fig. 10, there is provided a data analysis apparatus 01 including: an obtaining module 11, a processing module 12 and a sending module 13, wherein:
the acquisition module 11 is used for the client to acquire the log file corresponding to the software operation;
the processing module 12 is used for the client to perform integration processing on the abnormal data in the log file to obtain the abnormal data after the integration processing; the integration processing comprises merging the abnormal data and adding the effective field of the abnormal data; the valid field at least comprises a product to which the abnormal data belongs, an item to which the abnormal data belongs, a client ID and a server address;
the sending module 13 is used for the client to send the abnormal data after the integration processing to the server; the abnormal data is used for indicating the server to comprehensively analyze the abnormal data after the integration processing to obtain an analysis result; the comprehensive analysis at least comprises problem feature analysis, vulnerability and non-functional problem analysis and performance analysis of abnormal data.
In one embodiment, as shown in fig. 11, the processing module 12 includes an obtaining unit 121 and a processing unit 122, where:
an obtaining unit 121, configured to obtain, by the client, abnormal data of the log file according to the data format identifier in the log file;
and the processing unit 122 is configured to merge the abnormal data in the log file into a complete abnormal data by the client, and add an effective field of the abnormal data to obtain the abnormal data after the integration processing.
In one embodiment, as shown in fig. 12, the data analysis device 01 further includes a relay module 14;
the transfer module 14 is used for the client to send the abnormal data after the integration processing to the server through the transfer terminal; the transfer terminal is used for analyzing and packaging the abnormal data after the integration processing, and sending the abnormal data after the analysis and packaging processing to the server terminal; wherein, different client types correspond to different analysis and encapsulation processing modes.
In an embodiment, the transit module 14 is specifically configured to parse the integrated abnormal data, obtain a target field, and encapsulate the target field; the target field at least comprises the occurrence time of the abnormal data, the belonged product, the belonged item, the belonged class, the belonged method, the belonged row, the abnormal type and the address of the server.
In one embodiment, as shown in fig. 13, the data analysis device 01 further includes a verification module 15 and a connection module 16, wherein:
the verification module 15 is used for verifying the validity of the function configuration of the log file by the client;
and the connection module 16 is configured to establish network connection with the relay terminal if the function configuration is legal, and send the integrated abnormal data to the relay terminal.
In one embodiment, as shown in fig. 14, the data analysis apparatus 01 further includes a storage module 17;
the storage module 17 is used for the client to store the processing information of the log file in the database; the processing information of the log file at least comprises a file path, a file name, a file type, the line number of the last line in the log and a timestamp corresponding to the last line.
The implementation principle and technical effect of all the embodiments of the data analysis apparatus are similar to those of the embodiments corresponding to the data analysis method, and are not described herein again.
In one embodiment, as shown in fig. 15, the present embodiment provides another data analysis apparatus 02 including: a receiving module 21 and an analyzing module 22, wherein:
the receiving module 21 is configured to receive, by the server, the integrated abnormal data sent by the client, where the integrated abnormal data is obtained by integrating and processing the abnormal data in the log file corresponding to the software operation by the client; the integration processing comprises merging the abnormal data and adding the effective field of the abnormal data; the valid field at least comprises a product to which the abnormal data belongs, an item to which the abnormal data belongs, a client ID and a server address;
the analysis module 22 is used for the server side to perform comprehensive analysis on the abnormal data after the integration processing to obtain an analysis result; the comprehensive analysis at least comprises problem feature analysis, vulnerability and non-functional problem analysis and performance analysis of abnormal data.
In one embodiment, as shown in fig. 16, the analysis module 22 includes a verification unit 221 and an analysis unit 222, wherein:
the checking unit 221, configured to perform validity checking on the abnormal data after the integration processing by the server;
the analysis unit 222 is configured to perform a comprehensive analysis step on the abnormal data after the integration processing if the abnormal data after the integration processing is valid.
In one embodiment, as shown in fig. 17, the data analysis device 02 further includes a storage and display module 23;
and the storage and display module 23 is used for storing and displaying the analysis result by the server.
The implementation principle and technical effect of all the embodiments of the data analysis apparatus are similar to those of the embodiments corresponding to the data analysis method, and are not described herein again.
For specific limitations of the data analysis device, reference may be made to the above limitations of the data analysis method, which are not described herein again. The modules in the data analysis device can be wholly or partially realized by software, hardware and a combination thereof. The modules can be embedded in a hardware form or independent from a processor in the computer device, and can also be stored in a memory in the computer device in a software form, so that the processor can call and execute operations corresponding to the modules.
In one embodiment, a computer device, which may be a client or a server, is provided, and its internal structure diagram may be as shown in fig. 18. The computer device comprises a processor, a memory, a network interface, a database, a display screen and an input device which are connected through a system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device comprises a nonvolatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of an operating system and computer programs in the non-volatile storage medium. The network interface of the computer device is used for communicating with an external terminal through a network connection. The database of the computer device is used for storing data analysis data. The computer program is executed by a processor to implement a data analysis method. The display screen of the computer equipment can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer equipment can be a touch layer covered on the display screen, a key, a track ball or a touch pad arranged on the shell of the computer equipment, an external keyboard, a touch pad or a mouse and the like.
Those skilled in the art will appreciate that the architecture shown in fig. 18 is merely a block diagram of some of the structures associated with the disclosed aspects and is not intended to limit the computing devices to which the disclosed aspects apply, as particular computing devices may include more or less components than those shown, or may combine certain components, or have a different arrangement of components.
In one embodiment, a computer device is provided, comprising a memory and a processor, the memory having a computer program stored therein, the processor implementing the following steps when executing the computer program:
the client acquires a log file corresponding to software operation;
the client side integrates abnormal data in the log file to obtain integrated abnormal data; the integration processing comprises merging the abnormal data and adding the effective field of the abnormal data; the valid field at least comprises a product to which the abnormal data belongs, an item to which the abnormal data belongs, a client ID and a server address;
the client side sends the abnormal data after the integration processing to the server side; the abnormal data is used for indicating the server to comprehensively analyze the abnormal data after the integration processing to obtain an analysis result; the comprehensive analysis at least comprises problem feature analysis, vulnerability and non-functional problem analysis and performance analysis of abnormal data.
The implementation principle and technical effect of the computer device provided by the above embodiment are similar to those of the above method embodiment, and are not described herein again.
In one embodiment, a computer-readable storage medium is provided, having a computer program stored thereon, which when executed by a processor, performs the steps of:
the client acquires a log file corresponding to software operation;
the client side integrates abnormal data in the log file to obtain integrated abnormal data; the integration processing comprises merging the abnormal data and adding the effective field of the abnormal data; the valid field at least comprises a product to which the abnormal data belongs, an item to which the abnormal data belongs, a client ID and a server address;
the client side sends the abnormal data after the integration processing to the server side; the abnormal data is used for indicating the server to comprehensively analyze the abnormal data after the integration processing to obtain an analysis result; the comprehensive analysis at least comprises problem feature analysis, vulnerability and non-functional problem analysis and performance analysis of abnormal data.
The implementation principle and technical effect of the computer-readable storage medium provided by the above embodiments are similar to those of the above method embodiments, and are not described herein again.
It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by hardware instructions of a computer program, which can be stored in a non-volatile computer-readable storage medium, and when executed, can include the processes of the embodiments of the methods described above. Any reference to memory, storage, database, or other medium used in the embodiments provided herein may include non-volatile and/or volatile memory, among others. Non-volatile memory can include read-only memory (ROM), Programmable ROM (PROM), Electrically Programmable ROM (EPROM), Electrically Erasable Programmable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDRSDRAM), Enhanced SDRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), Rambus Direct RAM (RDRAM), direct bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).
The technical features of the above embodiments can be arbitrarily combined, and for the sake of brevity, all possible combinations of the technical features in the above embodiments are not described, but should be considered as the scope of the present specification as long as there is no contradiction between the combinations of the technical features.
The above-mentioned embodiments only express several embodiments of the present application, and the description thereof is more specific and detailed, but not construed as limiting the scope of the invention. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the concept of the present application, which falls within the scope of protection of the present application. Therefore, the protection scope of the present patent shall be subject to the appended claims.
Claims (14)
1. A method of data analysis, the method comprising:
the client acquires a log file corresponding to software operation;
the client integrates the abnormal data in the log file to obtain the integrated abnormal data; the integration processing comprises merging abnormal data and adding a valid field of the abnormal data; the valid field at least comprises a product to which the abnormal data belongs, an item to which the abnormal data belongs, the client ID and a server address;
the client side sends the abnormal data after the integration processing to a server side; the abnormal data are used for indicating the server to comprehensively analyze the abnormal data after the integration processing to obtain an analysis result; the comprehensive analysis at least comprises problem feature analysis, vulnerability and non-functional problem analysis and performance analysis of abnormal data.
2. The method of claim 1, wherein the step of integrating, by the client, the abnormal data in the log file to obtain the integrated abnormal data comprises:
the client acquires abnormal data of the log file according to the data format identifier in the log file;
and the client side combines all abnormal data in the abnormal data of the log file into a complete abnormal data respectively, and adds the effective field of the abnormal data to obtain the abnormal data after the integration processing.
3. The method according to claim 1 or 2, wherein the client sends the abnormal data after the integration processing to a server, and the method comprises:
the client sends the integrated abnormal data to the server through a transfer terminal; the transfer terminal is used for analyzing and packaging the abnormal data after the integration processing, and sending the abnormal data after the analysis and packaging processing to the server terminal; wherein, different client types correspond to different analysis and encapsulation processing modes.
4. The method of claim 3, wherein the parsing and encapsulating process comprises:
analyzing the abnormal data after the integration processing to obtain a target field, and packaging the target field; the target field at least comprises the occurrence time of the abnormal data, the belonged product, the belonged item, the belonged class, the belonged method, the belonged row, the abnormal type and the address of the server.
5. The method of claim 3, further comprising:
the client verifies the validity of the function configuration of the log file;
and if the function configuration is legal, establishing network connection with the transfer terminal, and sending the abnormal data after the integration processing to the transfer terminal.
6. The method according to claim 1, wherein after the client sends the abnormal data after the integration processing to a server, the method further comprises:
the client stores the processing information of the log file in a database; the processing information of the log file at least comprises a file path, a file name, a file type, the line number of the last line in the log and a timestamp corresponding to the last line.
7. A method of data analysis, the method comprising:
the server receives abnormal data after integration processing sent by the client, wherein the abnormal data after integration processing is data obtained by integrating and processing the abnormal data in a log file corresponding to software operation by the client; the integration processing comprises merging abnormal data and adding a valid field of the abnormal data; the valid field at least comprises a product to which the abnormal data belongs, an item to which the abnormal data belongs, the client ID and a server address;
the server comprehensively analyzes the abnormal data after the integration processing to obtain an analysis result; the comprehensive analysis at least comprises problem feature analysis, vulnerability and non-functional problem analysis and performance analysis of abnormal data.
8. The method of claim 7, wherein the comprehensive analysis of the abnormal data after the integration processing by the server includes:
the server checks the validity of the abnormal data after the integration processing;
and if the integrated abnormal data is valid, executing the step of comprehensively analyzing the integrated abnormal data.
9. The method of claim 7 or 8, wherein after said obtaining an analysis result, the method further comprises:
and the server stores and displays the analysis result.
10. A data analysis system, the system comprising:
a client for performing the steps of the method of any one of claims 1-6;
a server for performing the steps of the method of any one of claims 7-9.
11. A data analysis apparatus, characterized in that the apparatus comprises:
the acquisition module is used for acquiring a log file corresponding to software operation by a client;
the processing module is used for integrating the abnormal data in the log file by the client to obtain the integrated abnormal data; the integration processing comprises merging abnormal data and adding a valid field of the abnormal data; the valid field at least comprises a product to which the abnormal data belongs, an item to which the abnormal data belongs, the client ID and a server address;
the sending module is used for sending the abnormal data after the integration processing to a server side by the client side; the abnormal data are used for indicating the server to comprehensively analyze the abnormal data after the integration processing to obtain an analysis result; the comprehensive analysis at least comprises problem feature analysis, vulnerability and non-functional problem analysis and performance analysis of abnormal data.
12. A data analysis apparatus, characterized in that the apparatus comprises:
the receiving module is used for receiving the abnormal data after the integration processing sent by the client side by the server side, wherein the abnormal data after the integration processing is the data obtained by integrating and processing the abnormal data in the log file corresponding to the software operation by the client side; the integration processing comprises merging abnormal data and adding a valid field of the abnormal data; the valid field at least comprises a product to which the abnormal data belongs, an item to which the abnormal data belongs, the client ID and a server address;
the analysis module is used for comprehensively analyzing the abnormal data after the integration processing by the server side to obtain an analysis result; the comprehensive analysis at least comprises problem feature analysis, vulnerability and non-functional problem analysis and performance analysis of abnormal data.
13. A computer device comprising a memory and a processor, the memory storing a computer program, wherein the processor implements the steps of the method of any one of claims 1 to 9 when executing the computer program.
14. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the method of any one of claims 1 to 9.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911147117.1A CN111061628B (en) | 2019-11-21 | 2019-11-21 | Data analysis method, system, device, computer equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911147117.1A CN111061628B (en) | 2019-11-21 | 2019-11-21 | Data analysis method, system, device, computer equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111061628A true CN111061628A (en) | 2020-04-24 |
| CN111061628B CN111061628B (en) | 2023-09-01 |
Family
ID=70298734
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911147117.1A Active CN111061628B (en) | 2019-11-21 | 2019-11-21 | Data analysis method, system, device, computer equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111061628B (en) |
Cited By (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111931203A (en) * | 2020-07-15 | 2020-11-13 | 深信服科技股份有限公司 | Sensitive data analysis method, device, equipment and storage medium |
| CN111949551A (en) * | 2020-09-01 | 2020-11-17 | 网易（杭州）网络有限公司 | Application program testing method, device, equipment and storage medium |
| CN112395333A (en) * | 2020-11-20 | 2021-02-23 | 北京百度网讯科技有限公司 | Method and device for checking data exception, electronic equipment and storage medium |
| CN113127320A (en) * | 2021-04-08 | 2021-07-16 | 支付宝(杭州)信息技术有限公司 | Application program abnormity detection method, device, equipment and system |
| CN113850987A (en) * | 2020-12-11 | 2021-12-28 | 广东朝歌智慧互联科技有限公司 | System for detecting product quality |
| CN114238018A (en) * | 2021-12-17 | 2022-03-25 | 天翼爱音乐文化科技有限公司 | Method, system and device for detecting integrity of log collection file and storage medium |
| CN115065709A (en) * | 2022-03-10 | 2022-09-16 | 青岛海尔空调器有限总公司 | Information processing method, device and system |
| CN116366346A (en) * | 2023-04-04 | 2023-06-30 | 中国华能集团有限公司北京招标分公司 | DNS traffic reduction method |
| WO2023125773A1 (en) * | 2021-12-30 | 2023-07-06 | 网络通信与安全紫金山实验室 | Global exception handling method and platform in large-scale micro-service cluster scenario |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105828430A (en) * | 2015-01-08 | 2016-08-03 | 阿里巴巴集团控股有限公司 | Information acquisition and processing method, client and server |
| CN106649729A (en) * | 2016-12-23 | 2017-05-10 | 深圳市金证科技股份有限公司 | Log running method and system, client and server |
| CN106940679A (en) * | 2017-02-23 | 2017-07-11 | 中科创达软件股份有限公司 | Data processing method and device |
| CN108400893A (en) * | 2018-02-22 | 2018-08-14 | 北京奇艺世纪科技有限公司 | Processing method, device and the electronic equipment of journal file |
| CN108551404A (en) * | 2018-04-20 | 2018-09-18 | 北京百度网讯科技有限公司 | Method, apparatus, storage medium and the terminal device of client-side information analysis |
| CN109284269A (en) * | 2018-10-17 | 2019-01-29 | Oppo广东移动通信有限公司 | Abnormal log analysis method, device, storage medium and server |
| US20190079818A1 (en) * | 2017-09-08 | 2019-03-14 | Oracle International Corporation | Techniques for managing and analyzing log data |
| CN110059064A (en) * | 2019-03-20 | 2019-07-26 | 北京字节跳动网络技术有限公司 | Journal file processing method, device and computer readable storage medium |
-
2019
- 2019-11-21 CN CN201911147117.1A patent/CN111061628B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105828430A (en) * | 2015-01-08 | 2016-08-03 | 阿里巴巴集团控股有限公司 | Information acquisition and processing method, client and server |
| CN106649729A (en) * | 2016-12-23 | 2017-05-10 | 深圳市金证科技股份有限公司 | Log running method and system, client and server |
| CN106940679A (en) * | 2017-02-23 | 2017-07-11 | 中科创达软件股份有限公司 | Data processing method and device |
| US20190079818A1 (en) * | 2017-09-08 | 2019-03-14 | Oracle International Corporation | Techniques for managing and analyzing log data |
| CN108400893A (en) * | 2018-02-22 | 2018-08-14 | 北京奇艺世纪科技有限公司 | Processing method, device and the electronic equipment of journal file |
| CN108551404A (en) * | 2018-04-20 | 2018-09-18 | 北京百度网讯科技有限公司 | Method, apparatus, storage medium and the terminal device of client-side information analysis |
| CN109284269A (en) * | 2018-10-17 | 2019-01-29 | Oppo广东移动通信有限公司 | Abnormal log analysis method, device, storage medium and server |
| CN110059064A (en) * | 2019-03-20 | 2019-07-26 | 北京字节跳动网络技术有限公司 | Journal file processing method, device and computer readable storage medium |
Cited By (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111931203A (en) * | 2020-07-15 | 2020-11-13 | 深信服科技股份有限公司 | Sensitive data analysis method, device, equipment and storage medium |
| CN111949551A (en) * | 2020-09-01 | 2020-11-17 | 网易（杭州）网络有限公司 | Application program testing method, device, equipment and storage medium |
| CN112395333A (en) * | 2020-11-20 | 2021-02-23 | 北京百度网讯科技有限公司 | Method and device for checking data exception, electronic equipment and storage medium |
| CN112395333B (en) * | 2020-11-20 | 2023-07-25 | 北京百度网讯科技有限公司 | Method, device, electronic equipment and storage medium for checking data abnormality |
| CN113850987A (en) * | 2020-12-11 | 2021-12-28 | 广东朝歌智慧互联科技有限公司 | System for detecting product quality |
| CN113127320A (en) * | 2021-04-08 | 2021-07-16 | 支付宝(杭州)信息技术有限公司 | Application program abnormity detection method, device, equipment and system |
| CN114238018A (en) * | 2021-12-17 | 2022-03-25 | 天翼爱音乐文化科技有限公司 | Method, system and device for detecting integrity of log collection file and storage medium |
| WO2023125773A1 (en) * | 2021-12-30 | 2023-07-06 | 网络通信与安全紫金山实验室 | Global exception handling method and platform in large-scale micro-service cluster scenario |
| CN115065709A (en) * | 2022-03-10 | 2022-09-16 | 青岛海尔空调器有限总公司 | Information processing method, device and system |
| CN116366346A (en) * | 2023-04-04 | 2023-06-30 | 中国华能集团有限公司北京招标分公司 | DNS traffic reduction method |
| CN116366346B (en) * | 2023-04-04 | 2024-03-22 | 中国华能集团有限公司北京招标分公司 | DNS traffic reduction method |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111061628B (en) | 2023-09-01 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111061628B (en) | Data analysis method, system, device, computer equipment and storage medium | |
| CN109474578B (en) | Message checking method, device, computer equipment and storage medium | |
| CN110647438B (en) | Event monitoring method and device, computer equipment and storage medium | |
| US10769228B2 (en) | Systems and methods for web analytics testing and web development | |
| CN107766190B (en) | Interface testing method and device, computer equipment and storage medium | |
| CN110445637B (en) | Event monitoring method, system, computer device and storage medium | |
| CN111143163B (en) | Data monitoring method, device, computer equipment and storage medium | |
| CN112506894A (en) | Service chain log processing method and device based on link tracking and computer equipment | |
| CN111475370A (en) | Operation and maintenance monitoring method, device and equipment based on data center and storage medium | |
| CN111475376B (en) | Method, apparatus, computer device and storage medium for processing test data | |
| CN113315828B (en) | Traffic recording method and device, traffic recording equipment and storage medium | |
| CN109325058B (en) | Rule batch comparison method, device, computer equipment and storage medium | |
| CN112395177A (en) | Interactive processing method, device and equipment of service data and storage medium | |
| CN112269744A (en) | System abnormity testing method and device, computer equipment and storage medium | |
| CN111538754A (en) | Data collection management system, method, device, equipment and storage medium | |
| CN110177024B (en) | Monitoring method of hotspot equipment, client, server and system | |
| CN114201090A (en) | Product production abnormity early warning system, method, computer equipment and storage medium | |
| CN116069838A (en) | Data processing method, device, computer equipment and storage medium | |
| CN115422601A (en) | RPA robot process automation implementation method and system | |
| CN111459753B (en) | Point-of-sale terminal data processing method, device, computer equipment and storage medium | |
| CN115269424A (en) | Automatic regression testing method, device, equipment and storage medium for production flow | |
| CN111176930B (en) | Component operation data processing method and device, computer equipment and storage medium | |
| CN112416896A (en) | Data abnormity warning method and device, storage medium and electronic device | |
| CN111722994A (en) | Task request response method and device | |
| CN114840599B (en) | Semiconductor source data parsing method, ETL system, computer device and product |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20220117 Address after: Room 1423, No. 1256 and 1258, Wanrong Road, Jing'an District, Shanghai 200040 Applicant after: Tianyi Digital Life Technology Co.,Ltd. Address before: 1 / F and 2 / F, East Garden, Huatian International Plaza, 211 Longkou Middle Road, Tianhe District, Guangzhou, Guangdong 510630 Applicant before: Century Dragon Information Network Co.,Ltd. |
|
| GR01 | Patent grant | ||
| GR01 | Patent grant |