CN111858278A - Log analysis method and system based on big data processing and readable storage device - Google Patents

Log analysis method and system based on big data processing and readable storage device Download PDF

Info

Publication number
CN111858278A
CN111858278A CN202010651841.4A CN202010651841A CN111858278A CN 111858278 A CN111858278 A CN 111858278A CN 202010651841 A CN202010651841 A CN 202010651841A CN 111858278 A CN111858278 A CN 111858278A
Authority
CN
China
Prior art keywords
data
log
analysis method
processing
log analysis
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202010651841.4A
Other languages
Chinese (zh)
Inventor
杨勋
胡建国
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing Guolian Video Information Technology Co ltd
Original Assignee
Beijing Guolian Video Information Technology Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing Guolian Video Information Technology Co ltd filed Critical Beijing Guolian Video Information Technology Co ltd
Priority to CN202010651841.4A priority Critical patent/CN111858278A/en
Publication of CN111858278A publication Critical patent/CN111858278A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00Error detection; Error correction; Monitoring
    • G06F11/30Monitoring
    • G06F11/34Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
    • G06F11/3466Performance evaluation by tracing or monitoring
    • G06F11/3476Data logging
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00Error detection; Error correction; Monitoring
    • G06F11/30Monitoring
    • G06F11/34Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
    • G06F11/3438Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment monitoring of user actions
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/21Design, administration or maintenance of databases
    • G06F16/215Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/90Details of database functions independent of the retrieved data types
    • G06F16/95Retrieval from the web
    • G06F16/958Organisation or management of web site content, e.g. publishing, maintaining pages or automatic linking
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06QINFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q10/00Administration; Management
    • G06Q10/04Forecasting or optimisation specially adapted for administrative or management purposes, e.g. linear programming or "cutting stock problem"
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L41/00Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
    • H04L41/06Management of faults, events, alarms or notifications
    • H04L41/069Management of faults, events, alarms or notifications using logs of notifications; Post-processing of notifications

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • General Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Quality & Reliability (AREA)
  • Databases & Information Systems (AREA)
  • Business, Economics & Management (AREA)
  • Data Mining & Analysis (AREA)
  • Computer Hardware Design (AREA)
  • Economics (AREA)
  • Human Resources & Organizations (AREA)
  • Strategic Management (AREA)
  • Game Theory and Decision Science (AREA)
  • Marketing (AREA)
  • Operations Research (AREA)
  • Entrepreneurship & Innovation (AREA)
  • Tourism & Hospitality (AREA)
  • General Business, Economics & Management (AREA)
  • Development Economics (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Debugging And Monitoring (AREA)

Abstract

The invention provides a log analysis method, a system and a readable storage device based on big data processing, wherein the log analysis method comprises the following steps: s1: the log collection service BLS collects logs from a log source; s2: performing data cold treatment and data heat treatment on the log; s3: performing big data distributed calculation, processing, analysis and reporting through the BMR; s4: the data processed by the BMR is written into a data warehouse or the data in the BOS is combined with the machine learning BML to carry out user behavior prediction analysis operation, and the main source of the user, contents on favorite websites, loyalty of the user and the like can be seen through a log analysis system. By analyzing the user behavior log, the invention can further optimize the layout and the function of the website so as to improve the user experience and the like. And dividing the popularization budget and emphasizing and optimizing the tendency points of the user group and the like through the analysis result.

Description

Log analysis method and system based on big data processing and readable storage device
[ technical field ] A method for producing a semiconductor device
The invention relates to the technical field of big data processing, in particular to a log analysis method and system based on big data processing and a readable storage device.
[ background of the invention ]
In internet applications, logs are very important data, and since internet projects often require 7 × 24 uninterrupted operation, it is necessary to acquire log data related to the operation of a monitoring system and analyze the log data. The log analysis system is an integrated data environment oriented to analysis, and provides a strategic set of system data support for an enterprise decision making process. Through the analysis of the data in the data warehouse, the enterprise can be helped to improve the business process, control the cost, improve the product quality and the like. In the big data era, log data is a definite record of the operation condition of each company, organization or organization, and is one of the most direct, most easily available and most widely covered data sources of the use traces of users in the company product process. The collection and analysis of logs are often handled by techniques that employ large data.
Accordingly, there is a need to develop a log analysis method, system and readable storage device based on big data processing to address the shortcomings of the prior art, so as to solve or alleviate one or more of the above problems.
[ summary of the invention ]
In view of this, the present invention provides a log analysis method, a log analysis system and a readable storage device based on big data processing, which can see the main source of the user, which contents on the favorite website and the loyalty of the user, further optimize the layout and functions of the website, so as to improve the user experience.
In one aspect, the present invention provides a log analysis method, including the following steps:
s1: a log collection service, the BLS collecting logs from a log source;
s2: respectively carrying out data cold treatment and data heat treatment on the log;
s3: performing big data distributed calculation on the processed data through the BMR, and analyzing;
s4: writing the data after BMR calculation into a data warehouse or performing user behavior prediction analysis operation by combining data in BOS and machine learning BML;
s5: applying and displaying the analysis result, and providing an alarm for operation and maintenance personnel according to the result of the data heat treatment; and displaying the result of cold data processing through a BI tool.
As for the above-mentioned aspect and any possible implementation manner, there is further provided an implementation manner, where the data cold processing in S2 specifically is: and writing the log into an object storage BOS for storage or into an HBase cluster, and then accessing a Hive or Spark SQL cluster for analysis and processing.
The above-mentioned aspect and any possible implementation manner further provide an implementation manner, and the data heat treatment in S2 is specifically: and inputting the log into a message service Kafka as a message queue, delivering the log to a streaming computing BSC, performing real-time computing processing on the log data, and writing the processed data into the Kafka.
There is further provided in accordance with the above-described aspects and any possible implementations, an implementation in which the BMR is a fully hosted Hadoop/Spark cluster.
In the above-described aspect and any possible implementation manner, an implementation manner is further provided, where in the S1 is specifically that, by performing hosted log collection service through BLS, a user needs to configure a source address, a destination address, and a collection rule.
As to the above-mentioned aspect and any possible implementation manner, there is further provided an implementation manner, where the S3 specifically includes:
the S3 specifically includes:
s31: data cleaning, namely cleaning data by using a distributed computing frame, and storing the cleaned data in a data warehouse or reserving the cleaned data in the computing frame;
s32: and carrying out service statistics on the cleaned data by using Spark, Hive, MapReduce or Flink frames, and analyzing according to the content of the big data.
The above-described aspects and any possible implementations further provide an implementation in which the distributed computing framework includes Spark, Hive, and MapReduce.
The above-mentioned aspects and any possible implementation further provide an implementation, and the BI tool presentation result in S5 includes a pie chart, a bar chart, a map and a line chart.
The above-mentioned aspects and any possible implementation manners further provide a big data processing-based log analysis device, which includes
The log collection module is used for collecting logs from a log source;
the processing and analyzing module is used for carrying out data cleaning, cold treatment and heat treatment on the log, and storing and predicting and analyzing the processing result;
and the application display module is used for alarming according to the stored processing result and displaying the prediction analysis result in a BI tool.
The above-described aspects and any possible implementation further provide a computer-readable storage medium having stored thereon a processing program of a log analysis method, which when executed by a processor, implements the steps of the log analysis method according to any one of the above-described embodiments.
The above-mentioned aspects and any possible implementation manners further provide an application of a big data processing-based log analysis method, where the log analysis method is used for electronic banking, communication operators, or e-commerce operation platforms.
Compared with the prior art, the invention can obtain the following technical effects:
the system supports off-line and real-time processing of cold and hot data; and the method supports multiple data sources, the processing of complex data structures and the construction of complex portraits. The main source of the user, which contents on the favorite website, the loyalty of the user, etc. can be seen by the log analysis system. By analyzing the user behavior log, the layout and the functions of the website can be further optimized, so that the user experience is improved. And dividing the popularization budget and emphasizing and optimizing the tendency points of the user group and the like through the analysis result.
Of course, it is not necessary for any one product in which the invention is practiced to achieve all of the above-described technical effects simultaneously.
[ description of the drawings ]
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings needed to be used in the embodiments will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.
Fig. 1 is a flowchart of a log analysis method according to an embodiment of the present invention.
[ detailed description ] embodiments
For better understanding of the technical solutions of the present invention, the following detailed descriptions of the embodiments of the present invention are provided with reference to the accompanying drawings.
It should be understood that the described embodiments are only some embodiments of the invention, and not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The terminology used in the embodiments of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used in the examples of the present invention and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.
The invention provides a log analysis method based on big data processing, as shown in fig. 1, the log analysis method comprises the following steps:
s1: the log collection service BLS collects logs from a log source;
s2: performing data cold treatment and data heat treatment on the log;
s3: performing big data distributed calculation, processing, analysis and reporting through the BMR;
s4: writing the data processed by the BMR into a data warehouse or performing user behavior prediction analysis operation by combining the data in the BOS and the machine learning BML;
s5: applying and displaying the analysis result, and providing an alarm for operation and maintenance personnel according to the result of the data heat treatment; the cold data was presented by the BI tool.
The data cold processing in the step S2 specifically includes: and writing the log into an object storage BOS for storage or into an HBase cluster, and then accessing a Hive or spark SQL cluster for analysis and processing.
The data heat treatment in the S2 is specifically as follows: and the log access message service Kafka is used as a message queue, the message queue is delivered to a streaming computing BSC to perform real-time computing processing on the log data, and the processed data is written into the Kafka.
The BMR is a fully hosted Hadoop/Spark cluster, and the S1 is specifically that a user needs to configure a source address, a destination address and a collection rule by performing hosted log collection service through BLS.
The S3 specifically includes:
s31: data cleaning, namely cleaning data by using a distributed computing frame, and storing the cleaned data in a data warehouse or reserving the cleaned data in the computing frame;
s32: and carrying out service statistics on the cleaned data by using Spark, Hive, MapReduce or Flink frames, and analyzing according to the content of the big data.
The distributed computing framework includes Spark, Hive, and MapReduce. The BI tool display result in S5 includes a pie chart, a bar chart, a map, and a line chart.
A log analysis device based on big data processing comprises
The log collection module is used for collecting logs from a log source;
the processing and analyzing module is used for carrying out data cleaning, cold treatment and heat treatment on the log, and storing and predicting and analyzing the processing result;
and the application display module is used for alarming according to the stored processing result and displaying the prediction analysis result in a BI tool.
A computer-readable storage medium, on which a processing program of a log analysis is stored, which, when executed by a processor, implements the steps of the log analysis method according to any one of the preceding claims.
The invention aims at the problem that the user increasingly depends on the communication service provided by the operator in daily life, so that the operator retains a large amount of user data. Activating the part of data potential value based on big data. Operators desire to mine hidden information values in specific directions from the retained user log data, such as: user behavior prediction, marketing result monitoring, credit risk prevention and control, real estate addressing, population migration distribution, transportation planning, commercial public opinion and other information. Aiming at the pain point of the operator, the invention provides data mining analysis service based on the log.
Under the condition of ensuring the privacy of the user, the log data of the operator is used, and a DPI system, a charging system and other data of the operator are combined. Hundreds of primary dimension classifications such as user habits, activity paths and consumption scenes and nearly thousands of secondary dimension classifications are constructed and used for constructing user portraits, and different time granularities of different dimensions of the user portraits are calibrated in a specific mode, so that the user portraits are gradually enriched. Ensuring a precise degree of provision of a particular user-specific service.
The working principle of the invention is as follows:
the log analysis system for big data processing is divided into five modules.
Data acquisition: data was collected using flash, and web logs were written to the HDFS of the BLS.
Data cleaning: the cleaned data is stored in a data warehouse or Hive, sparkSQL by using Spark, Hive, MapReduce or other distributed computing frameworks.
Data processing: and (4) carrying out statistics and analysis on corresponding services as required (using frames such as Spark, Hive, MapReduce, Flink and the like).
And (3) data processing result storage: and storing the result in a database such as RDBMS (relational database management system), NoSQL (structured query language) and the like.
Visualization of data: and the following are shown in a graphical display mode: pie charts, bar charts, maps, line graphs.
The system supports off-line and real-time processing of cold and hot data; and the method supports multiple data sources, the processing of complex data structures and the construction of complex portraits. The main source of the user, which contents on the favorite website, the loyalty of the user, etc. can be seen by the log analysis system. By analyzing the user behavior log, the invention can further optimize the layout and the function of the website so as to improve the user experience and the like. And dividing the popularization budget and emphasizing and optimizing the tendency points of the user group and the like through the analysis result.
The log analysis system and method based on big data processing simultaneously realize hot data processing and cold data processing, and comprise three modules of log collection, processing and analysis, application and display.
At the log collection module, the log collection service BLS collects logs from a log source (e.g., a server). The BLS is a hosted log collection service, and a user can realize high-reliability and high-availability collection of logs only by configuring simple information such as a source address, a destination address, a collection rule and the like.
The collected logs can be accessed to the log processing module. On one hand, for a thermal data processing scene, the log access message service Kafka can be used as a message queue, the message queue is delivered to the streaming computing BSC to perform real-time computing processing on the log data, and then the processed data is written into the Kafka. On the other hand, for a cold data processing scene, the log can be written into the object storage BOS for storage, or directly written into the HBase cluster, and then accessed into the Hive and spark SQL clusters for analysis and processing. The BMR is a fully hosted Hadoop/Spark cluster, and is focused on big data processing, analysis and reporting by means of a big data distributed computing technology. The data processed by the BMR can be written into a data warehouse. Meanwhile, analysis operations such as user behavior prediction can be directly carried out by combining data in the BOS and the machine learning BML.
In the application and display module, the thermal data can provide an alarm to operation and maintenance personnel after being processed; the cold data may be presented by the BI tool.
The log analysis method can be used for banks, communication operators and e-commerce operation platforms.
The log analysis method, system and readable storage device based on big data processing provided by the embodiments of the present application are described in detail above. The above description of the embodiments is only for the purpose of helping to understand the method of the present application and its core ideas; meanwhile, for a person skilled in the art, according to the idea of the present application, there may be variations in the specific embodiments and the application scope, and in summary, the content of the present specification should not be construed as a limitation to the present application.
As used in the specification and claims, certain terms are used to refer to particular components. As one skilled in the art will appreciate, manufacturers may refer to a component by different names. This specification and claims do not intend to distinguish between components that differ in name but not function. In the following description and in the claims, the terms "include" and "comprise" are used in an open-ended fashion, and thus should be interpreted to mean "include, but not limited to. "substantially" means within an acceptable error range, and a person skilled in the art can solve the technical problem within a certain error range to substantially achieve the technical effect. The description which follows is a preferred embodiment of the present application, but is made for the purpose of illustrating the general principles of the application and not for the purpose of limiting the scope of the application. The protection scope of the present application shall be subject to the definitions of the appended claims.
It is also noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a good or system that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such good or system. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other like elements in a commodity or system that includes the element.
It should be understood that the term "and/or" as used herein is merely one type of association that describes an associated object, meaning that three relationships may exist, e.g., a and/or B may mean: a exists alone, A and B exist simultaneously, and B exists alone. In addition, the character "/" herein generally indicates that the former and latter related objects are in an "or" relationship.
The foregoing description shows and describes several preferred embodiments of the present application, but as aforementioned, it is to be understood that the application is not limited to the forms disclosed herein, but is not to be construed as excluding other embodiments and is capable of use in various other combinations, modifications, and environments and is capable of changes within the scope of the application as described herein, commensurate with the above teachings, or the skill or knowledge of the relevant art. And that modifications and variations may be effected by those skilled in the art without departing from the spirit and scope of the application, which is to be protected by the claims appended hereto.

Claims (10)

1. A log analysis method based on big data processing is characterized by comprising the following steps:
s1: a log collection service, the BLS collecting logs from a log source;
s2: respectively carrying out data cold treatment and data heat treatment on the log;
s3: performing big data distributed calculation on the processed data through the BMR, and analyzing;
s4: writing the data after BMR calculation into a data warehouse or performing user behavior prediction analysis operation by combining data in BOS and machine learning BML;
s5: applying and displaying the analysis result, and providing an alarm for operation and maintenance personnel according to the result of the data heat treatment; and displaying the result of cold data processing through a BI tool.
2. The log analysis method according to claim 1, wherein the data cold processing in S2 is specifically: and writing the log into an object storage BOS for storage or into an HBase cluster, and then accessing a Hive or Spark SQL cluster for analysis and processing.
3. The log analysis method according to claim 1, wherein the data heat treatment in S2 is specifically: and inputting the log into a message service Kafka as a message queue, delivering the log to a streaming computing BSC, performing real-time computing processing on the log data, and writing the processed data into the Kafka.
4. The log analysis method as claimed in claim 1, wherein the S1 is implemented by a hosted log collection service through BLS, and the user needs to configure the source address, the destination address and the collection rule.
5. The log analysis method according to claim 1, wherein the S3 specifically includes:
s31: data cleaning, namely cleaning data by using a distributed computing frame, and storing the cleaned data in a data warehouse or reserving the cleaned data in the computing frame;
s32: and carrying out service statistics on the cleaned data by using Spark, Hive, MapReduce or Flink frames, and analyzing according to the content of the big data.
6. The log analysis method of claim 1, wherein the distributed computing framework comprises Spark, Hive, and MapReduce.
7. The log analysis method as claimed in claim 1, wherein the BI tool presentation result in S5 includes a pie chart, a bar chart, a map and a line chart.
8. A log analysis device based on big data processing, comprising the log analysis method of any of the above claims 1 to 7, wherein the device comprises
The log collection module is used for collecting logs from a log source;
The processing and analyzing module is used for carrying out data cleaning, cold treatment and heat treatment on the log, and storing and predicting and analyzing the processing result;
and the application display module is used for alarming according to the stored processing result and displaying the prediction analysis result in a BI tool.
9. A computer-readable storage medium, characterized in that the computer-readable storage medium has stored thereon a processing program of a log analysis method, which when executed by a processor implements the steps of the log analysis method according to any one of claims 1 to 7.
10. Application of a log analysis method based on big data processing, based on the steps of the log analysis method of any one of claims 1 to 7, wherein the log analysis method is used for electronic banking, communication operators or e-commerce operation platforms.
CN202010651841.4A 2020-07-08 2020-07-08 Log analysis method and system based on big data processing and readable storage device Pending CN111858278A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202010651841.4A CN111858278A (en) 2020-07-08 2020-07-08 Log analysis method and system based on big data processing and readable storage device

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202010651841.4A CN111858278A (en) 2020-07-08 2020-07-08 Log analysis method and system based on big data processing and readable storage device

Publications (1)

Publication Number Publication Date
CN111858278A true CN111858278A (en) 2020-10-30

Family

ID=73153153

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202010651841.4A Pending CN111858278A (en) 2020-07-08 2020-07-08 Log analysis method and system based on big data processing and readable storage device

Country Status (1)

Country Link
CN (1) CN111858278A (en)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113094250A (en) * 2021-05-12 2021-07-09 成都新希望金融信息有限公司 Log early warning method and device, electronic equipment and storage medium
CN113518365A (en) * 2021-04-29 2021-10-19 北京红山信息科技研究院有限公司 Data association method, device, server and storage medium
CN113836431A (en) * 2021-10-19 2021-12-24 中国平安人寿保险股份有限公司 User recommendation method, device, equipment and medium based on user duration

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2017071134A1 (en) * 2015-10-28 2017-05-04 北京汇商融通信息技术有限公司 Distributed tracking system
CN107577805A (en) * 2017-09-26 2018-01-12 华南理工大学 A kind of business service system towards the analysis of daily record big data
CN110489453A (en) * 2019-07-02 2019-11-22 广东工业大学 User's game real-time recommendation method and system based on big data log analysis
CN110690984A (en) * 2018-07-05 2020-01-14 上海宝信软件股份有限公司 Spark-based big data weblog acquisition, analysis and early warning method and system
US20200160230A1 (en) * 2018-11-19 2020-05-21 International Business Machines Corporation Tool-specific alerting rules based on abnormal and normal patterns obtained from history logs

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2017071134A1 (en) * 2015-10-28 2017-05-04 北京汇商融通信息技术有限公司 Distributed tracking system
CN107577805A (en) * 2017-09-26 2018-01-12 华南理工大学 A kind of business service system towards the analysis of daily record big data
CN110690984A (en) * 2018-07-05 2020-01-14 上海宝信软件股份有限公司 Spark-based big data weblog acquisition, analysis and early warning method and system
US20200160230A1 (en) * 2018-11-19 2020-05-21 International Business Machines Corporation Tool-specific alerting rules based on abnormal and normal patterns obtained from history logs
CN110489453A (en) * 2019-07-02 2019-11-22 广东工业大学 User's game real-time recommendation method and system based on big data log analysis

Non-Patent Citations (3)

* Cited by examiner, † Cited by third party
Title
杨潇黎;蒋廷耀;金鑫;罗神;: "分布式web日志处理平台的研究与实现", 信息通信, no. 03 *
田晓旭: "【技术分享】百度开放云张琪:大数据时代的数据仓储", pages 4 - 13, Retrieved from the Internet <URL:https://mp.weixin.qq.com/s/iHEAYUqEG5RXdwaOOnrjdA> *
马延超;王超;李尚同;: "基于大数据技术的日志统计与分析系统研究", 电脑知识与技术, no. 34 *

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113518365A (en) * 2021-04-29 2021-10-19 北京红山信息科技研究院有限公司 Data association method, device, server and storage medium
CN113518365B (en) * 2021-04-29 2023-11-17 北京红山信息科技研究院有限公司 Data association method, device, server and storage medium
CN113094250A (en) * 2021-05-12 2021-07-09 成都新希望金融信息有限公司 Log early warning method and device, electronic equipment and storage medium
CN113094250B (en) * 2021-05-12 2023-08-18 成都新希望金融信息有限公司 Log early warning method and device, electronic equipment and storage medium
CN113836431A (en) * 2021-10-19 2021-12-24 中国平安人寿保险股份有限公司 User recommendation method, device, equipment and medium based on user duration

Similar Documents

Publication Publication Date Title
US10235430B2 (en) Systems, methods, and apparatuses for detecting activity patterns
CN108509485B (en) Data preprocessing method and device, computer equipment and storage medium
US9009850B2 (en) Database management by analyzing usage of database fields
CN111858278A (en) Log analysis method and system based on big data processing and readable storage device
CN107315776A (en) A kind of data management system based on cloud computing
CN103678647A (en) Method and system for recommending information
US9910858B2 (en) System and method for providing contextual analytics data
US20170337568A1 (en) Data processing method, system and computer device based on electronic payment behaviors
Höpken et al. Tourism knowledge destination
US20170046376A1 (en) Method and system for monitoring data quality and dependency
Imran et al. Data provenance
Pospiech et al. A descriptive big data model using grounded theory
Altalhi et al. Developing a framework and algorithm for scalability to evaluate the performance and throughput of CRM systems
CN108170561B (en) Disaster recovery backup method, device and system
Hogan Data center
US20150081356A1 (en) Dynamic multi-dimensional business reports
US9361379B1 (en) Systems and methods providing recommendation data
CN111143328A (en) Agile business intelligent data construction method, system, equipment and storage medium
US20160283948A1 (en) End user trend identification to identify information gaps
Wen Data sharing
Zhang Data Synthesis
Upadhyay et al. The major boom of an Unstructured Big Data on Social Media
CN114253951B (en) Data processing method, system and second server
Chen Digital Ecosystem
US9218437B1 (en) Systems and methods providing event data

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination