CN108334556A - A kind of method and system of analysis internet finance massive logs - Google Patents
A kind of method and system of analysis internet finance massive logs Download PDFInfo
- Publication number
- CN108334556A CN108334556A CN201711495499.8A CN201711495499A CN108334556A CN 108334556 A CN108334556 A CN 108334556A CN 201711495499 A CN201711495499 A CN 201711495499A CN 108334556 A CN108334556 A CN 108334556A
- Authority
- CN
- China
- Prior art keywords
- real
- time
- business diary
- massive logs
- internet finance
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/16—File or folder operations, e.g. details of user interfaces specifically adapted to file systems
- G06F16/168—Details of user interfaces specifically adapted to file systems, e.g. browsing and visualisation, 2d or 3d GUIs
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/1805—Append-only file systems, e.g. using logs or journals to store data
- G06F16/1815—Journaling file systems
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Human Computer Interaction (AREA)
- Debugging And Monitoring (AREA)
Abstract
The invention discloses a kind of systems and analysis method of analysis internet finance massive logs, which includes the acquisition module being linked in sequence, transmission module and processing module；Wherein, acquisition module utilizes flume real-time collecting business diaries；Transmission module is transmitted business diary using kafka technologies, and processing module receives business diary and to business diary calculate and analyzed to obtain monitoring data in real time using storm.By flume come real-time collecting business diary, the real-time collecting to massive logs is realized；Also by kafka transmission services daily records, ensure that when there is high concurrent to be not in performance bottleneck；And by the horizontal expansion capability of storm, mass data can be handled, and real-time is high.
Description
Technical field
The present invention relates to internet financial technology fields, it particularly relates to a kind of point of internet finance massive logs
Analysis system and analysis method.
Background technology
Existing log system is partly to be directed to performance evaluation, is partly to be directed to data processing.And at data
Reason, be all off-line data processing, it is difficult to accomplish to monitor in real time, and is difficult the application scenarios for adapting to intermittent high concurrent, especially
It is this to data consistency for internet finance, and timeliness requires high application, if event on line cannot be directed in time
Barrier responds, and the influence brought is just very big.In addition to this, existing log system cannot quickly handle mass data, lead
Cause problem is found very slow.
Generally, existing log system has the following defects：
One, intermittent high concurrent situation cannot effectively be coped with.The Internet, applications are held because of the reasons such as period, operation means
Easily there is some period high concurrent situation, this when of daily record amount can explode, and it is easy to appear the feelings of delay machine if processing is bad
Condition.For example, some unexpected operation activities cause on line, traffic pressure is big, daily record amount blowout, and holding makes log system delay machine very much；
Two, real-time is poor.The much all only off-line analysis daily records, or analysis daily record quasi real time of existing log system, when
Occur quickly and effectively responding when problem on line；
Three, the data volume handled is limited.The data volume of existing log processing is all than relatively limited, when this leads to that long enough cannot be preserved
Between daily record, be unfavorable for problem investigation.
For the problems in the relevant technologies, currently no effective solution has been proposed.
Invention content
For the above problem in the related technology, the present invention propose it is a kind of analysis internet finance massive logs system and
Analysis method can efficiently solve the risk that high concurrent is brought using kafka as log transmission tool；Using storm collection
Group's real-time data collection, analysis result, realize massive logs and analyze in real time.
The technical proposal of the invention is realized in this way：
According to an aspect of the invention, there is provided a kind of system of analysis internet finance massive logs, which is characterized in that
Including acquisition module, transmission module and the processing module being linked in sequence；Wherein, acquisition module utilizes flume real-time collecting industry
Business daily record；Transmission module is transmitted business diary using kafka technologies, and processing module receives business diary and utilizes
Storm to business diary calculate and analyzed to obtain monitoring data in real time.
In one embodiment, further include：Memory module is connected to processing module；Wherein, memory module includes redis
Storage unit, wherein redis storage units are used for storage statistical data.
In one embodiment, memory module further includes：MySQL storage units and Elasticsearch storages are single
Member；Wherein, MySQL storage units are used for storage rule data；And Elasticsearch storage units are for storing daily record text
Notebook data.
In one embodiment, processing module includes：Alarm control unit, for according to the result for calculating and analyzing in real time
Generate alarm feedback.
In one embodiment, further include web display modules, be connected to memory module, the real-time dynamic exhibition of web display modules
Show monitoring data.
According to another aspect of the present invention, a kind of method of analysis internet finance massive logs is provided, feature exists
In, including：
Utilize flume real-time collecting business diaries；
Business diary is transmitted using kafka technologies；
It receives business diary and business diary calculate and analyzed to obtain monitoring data in real time using storm.
In one embodiment, further include after to business diary calculate and analyze in real time：Storage statistical data.
In one embodiment, further include after to business diary calculate and analyze in real time：Storage rule data；
And storage daily record text data.
In one embodiment, further include after to business diary calculate and analyze in real time：It is calculated according to real-time
Alarm feedback is generated with the result of analysis.
In one embodiment, further include after to business diary calculate and analyze in real time：Real-time Dynamic Display
Monitoring data.
The present invention, come real-time collecting business diary, realizes the real-time collecting to massive logs by flume；Also pass through
Kafka transmission services daily records ensure that when there is high concurrent to be not in performance bottleneck；And it is expanded by the level of storm
Ability can handle mass data, and real-time is high.
Description of the drawings
It in order to more clearly explain the embodiment of the invention or the technical proposal in the existing technology, below will be to institute in embodiment
Attached drawing to be used is needed to be briefly described, it should be apparent that, the accompanying drawings in the following description is only some implementations of the present invention
Example, for those of ordinary skill in the art, without creative efforts, can also obtain according to these attached drawings
Obtain other attached drawings.
Fig. 1 is a kind of schematic diagram of the system of analysis internet finance massive logs according to the ... of the embodiment of the present invention；
Fig. 2 is a kind of flow chart of the method for analysis internet finance massive logs according to the ... of the embodiment of the present invention.
Specific implementation mode
Following will be combined with the drawings in the embodiments of the present invention, and technical solution in the embodiment of the present invention carries out clear, complete
Site preparation describes, it is clear that described embodiments are only a part of the embodiments of the present invention, instead of all the embodiments.It is based on
Embodiment in the present invention, the every other embodiment that those of ordinary skill in the art are obtained belong to what the present invention protected
Range.
Such as Fig. 1, a kind of signal of the system of analysis internet finance massive logs according to the ... of the embodiment of the present invention is shown
Figure, analysis system 100 according to the ... of the embodiment of the present invention include：Acquisition module 10, transmission module 20 and the place being linked in sequence
Manage module 30；
Wherein, acquisition module 10 utilizes flume real-time collecting business diaries；Transmission module 20 is using kafka technologies to business day
Will is transmitted, and processing module 30 receives business diary and to business diary calculate and analyze in real time to obtain using storm
To monitoring data.
Above-mentioned technical proposal realizes the real-time collecting to massive logs by flume come real-time collecting business diary；
Also by kafka transmission services daily records, ensure that when there is high concurrent to be not in performance bottleneck；And the water for passing through storm
Flat expansion capability can handle mass data, and real-time is high.
Specifically, flume is a distribution, reliable high performance tools, can be used for collecting from different data sources, is poly-
It closes, transmit a large amount of daily record datas to a central data source.The analysis system 100 of the present invention is distributed by flume real-time collectings
Daily record in each business application system realizes daily record distribution real-time collecting；Kafka is a kind of distribution of high-throughput
Formula distribution subscription message system, by using kafka as log transmission tool, Kafka handling capacities are big, and day not easy to lose
Will, the risk that effective solution high concurrent is brought, the disaster tolerance performance for improving high concurrent；Massive logs are analyzed in real time.It is above-mentioned
Storm can be the real-time computing clusters of storm, and real-time data collection and number can be analyzed in real time by using storm clusters
According to.
In one embodiment, as shown in Figure 1, analysis system 100 further includes：Memory module 40, is connected to processing module
30；Memory module 40 includes redis storage units 42, and wherein redis storage units 42 are used for storage statistical data.
Wherein, memory module 40 further includes：MySQL storage units 41 and Elasticsearch storage units 43；
MySQL storage units 41 are used for storage rule data；And Elasticsearch storage units 43 are for storing daily record textual data
According to.
In one embodiment, processing module 30 includes：Alarm control unit 33 is calculated and is analyzed in real time for basis
As a result alarm feedback is generated.Advantage is handled by using the stream data of storm, the daily record just generated can be handled in real time, and
The operating status for timely feedbacking application system on line can generate alarm feedback in time to the problem on line.It is i.e. of the invention
Analysis system 100 can provide alarm feedback in real time to the problem on line, and then it is convenient timely and effectively on line therefore
Barrier is handled.As shown in Figure 1, in some optional embodiments, processing module 30 can also specifically include：Log read list
Member 31, sampling statistics unit 32, samples storage unit 34 and result output unit 35.
In one embodiment, analysis system 100 further includes web display modules 50, is connected to memory module 40, web exhibitions
Show 50 real-time Dynamic Display monitoring data of module.According to actual demand, can be analyzed by 50 real-time exhibition of web display modules
As a result, monitoring data etc..
As shown in Figure 1, in some optional embodiments, web display modules 50 may include：Monitoring rules administrative unit
51, alarm channel management unit 52, performance parameter Dynamic Display unit 53, alarm object management unit 54, alarm history displaying
Unit 55, service exception Dynamic Display unit 56, key words content retrieval unit 57.So as to carry out phase by website web
The function management answered and content displaying.
Shown in Fig. 1, the course of work of the analysis system 100 of the embodiment of the present invention is illustrated.Acquire mould
Block 10 using flume from application system on each line (application system 1 ..., application system n) real-time collecting business diaries, then
Transmission module 20 sends daily record using Kafka as log transmission tool, and Storm clusters receive the daily record that Kafka is sent, and
It is calculated in real time according to scheduled monitoring rules.And store statistical data to redis storage units 42, meet certain condition
MySQL storage units 41 are stored, it will be in the text data store of daily record to Elasticsearch storage units 43.Finally lead to
Web display modules 50 are crossed in the real-time Dynamic Display monitoring data in website.
In an alternative embodiment, the software environment of analysis system 100 of the invention is (SuSE) Linux OS；Firmly
Part environment is：Interconnected by express network including more and computer/server, with this may be implemented it is distributed, can parallel computation；
Programming language is java.It should be appreciated that software environment, hardware environment and the programming language of the analysis system 100 of the present invention can be with
It is configured as needed, the present invention does not make this restriction.
As shown in Fig. 2, according to an embodiment of the invention, additionally providing a kind of method of analysis internet finance massive logs
200, include the following steps：
S210 utilizes flume real-time collecting business diaries；
S220 is transmitted business diary using kafka technologies；
S230 receives business diary and to business diary calculate and analyzed to obtain monitoring data in real time using storm.
Wherein, can also include step after S230：S241, storage statistical data.
In one embodiment, can also include step after S230：
S242, storage rule data；And
S243 stores daily record text data.
In one embodiment, can also include step after S230：S244, according to the real-time result calculated and analyze
Generate alarm feedback.
In one embodiment, can also include step after S230：S250, real-time Dynamic Display monitoring data.
In conclusion the present invention by flume come real-time collecting business diary, realize the real-time receipts to massive logs
Collection；By kafka transmission services daily records, ensure that when there is high concurrent to be not in performance bottleneck；And pass through storm clusters
Real-time data collection can analyze data, real-time exhibition analysis result, provide alarm feedback in real time in real time.
The foregoing is merely illustrative of the preferred embodiments of the present invention, is not intended to limit the invention, all essences in the present invention
With within principle, any modification, equivalent replacement, improvement and so on should all be included in the protection scope of the present invention god.
Claims (10)
1. it is a kind of analysis internet finance massive logs system, which is characterized in that including be linked in sequence acquisition module, transmission
Module and processing module；
Wherein, the acquisition module utilizes flume real-time collecting business diaries；The transmission module is using kafka technologies to institute
It states business diary to be transmitted, the processing module is received the business diary and carries out the business diary using storm
It calculates and is analyzed to obtain monitoring data in real time.
2. a kind of system of analysis internet finance massive logs according to claim 1, which is characterized in that further include：
Memory module is connected to the processing module；
Wherein, the memory module includes redis storage units, wherein the redis storage units are used for storage statistical data.
3. a kind of system of analysis internet finance massive logs according to claim 2, which is characterized in that the storage
Module further includes：MySQL storage units and Elasticsearch storage units；
Wherein, the MySQL storage units are used for storage rule data；And
The Elasticsearch storage units are for storing daily record text data.
4. a kind of system of analysis internet finance massive logs according to claim 1, which is characterized in that the processing
Module includes：
Alarm control unit, for generating alarm feedback according to the result for calculating and analyzing in real time.
5. a kind of system of analysis internet finance massive logs according to claim 2, which is characterized in that further include
Web display modules, are connected to the memory module, monitoring data described in the real-time Dynamic Display of web display modules.
6. a kind of method of analysis internet finance massive logs, which is characterized in that including：
Utilize flume real-time collecting business diaries；
The business diary is transmitted using kafka technologies；
It receives the business diary and the business diary calculate and analyze in real time to obtain monitoring number using storm
According to.
7. a kind of method of analysis internet finance massive logs according to claim 6, which is characterized in that described
Business diary calculate in real time and further includes after analyzing：
Storage statistical data.
8. a kind of method of analysis internet finance massive logs according to claim 7, which is characterized in that described
Business diary calculate in real time and further includes after analyzing：
Storage rule data；And
Store daily record text data.
9. a kind of method of analysis internet finance massive logs according to claim 6, which is characterized in that described
Business diary calculate in real time and further includes after analyzing：
Alarm feedback is generated according to the real-time result for calculating and analyzing.
10. a kind of method of analysis internet finance massive logs according to claim 7, which is characterized in that institute
Business diary is stated calculate in real time and further include after analyzing：
Monitoring data described in real-time Dynamic Display.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711495499.8A CN108334556A (en) | 2017-12-31 | 2017-12-31 | A kind of method and system of analysis internet finance massive logs |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711495499.8A CN108334556A (en) | 2017-12-31 | 2017-12-31 | A kind of method and system of analysis internet finance massive logs |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN108334556A true CN108334556A (en) | 2018-07-27 |
Family
ID=62924032
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201711495499.8A Pending CN108334556A (en) | 2017-12-31 | 2017-12-31 | A kind of method and system of analysis internet finance massive logs |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108334556A (en) |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109165863A (en) * | 2018-09-06 | 2019-01-08 | 华泰证券股份有限公司 | Method, system and application based on the real-time valuation of distributed traffic fund asset |
| CN109726074A (en) * | 2018-08-31 | 2019-05-07 | 网联清算有限公司 | Log processing method, device, computer equipment and storage medium |
| CN111090646A (en) * | 2019-10-21 | 2020-05-01 | 中国科学院信息工程研究所 | Electromagnetic data processing platform |
| CN112651840A (en) * | 2021-01-13 | 2021-04-13 | 李青 | Business data log processing method and system based on block chain and digital finance |
| CN111858274B (en) * | 2020-07-02 | 2021-06-01 | 北京睿知图远科技有限公司 | Stability monitoring method for big data scoring system |
| CN113434376A (en) * | 2021-06-24 | 2021-09-24 | 山东浪潮科学研究院有限公司 | Web log analysis method and device based on NoSQL |
| CN115801353A (en) * | 2022-11-03 | 2023-03-14 | 智网安云(武汉)信息技术有限公司 | Linkage script processing method after real-time aggregation of safety event logs based on big data level |
-
2017
- 2017-12-31 CN CN201711495499.8A patent/CN108334556A/en active Pending
Cited By (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109726074A (en) * | 2018-08-31 | 2019-05-07 | 网联清算有限公司 | Log processing method, device, computer equipment and storage medium |
| CN109165863A (en) * | 2018-09-06 | 2019-01-08 | 华泰证券股份有限公司 | Method, system and application based on the real-time valuation of distributed traffic fund asset |
| CN109165863B (en) * | 2018-09-06 | 2022-03-22 | 华泰证券股份有限公司 | Method, system and application for real-time valuation of fund assets based on distributed data stream |
| CN111090646A (en) * | 2019-10-21 | 2020-05-01 | 中国科学院信息工程研究所 | Electromagnetic data processing platform |
| CN111090646B (en) * | 2019-10-21 | 2023-07-28 | 中国科学院信息工程研究所 | Electromagnetic data processing platform |
| CN111858274B (en) * | 2020-07-02 | 2021-06-01 | 北京睿知图远科技有限公司 | Stability monitoring method for big data scoring system |
| CN112651840A (en) * | 2021-01-13 | 2021-04-13 | 李青 | Business data log processing method and system based on block chain and digital finance |
| CN112651840B (en) * | 2021-01-13 | 2023-11-14 | 李青 | Business data log processing method and system based on blockchain and digital finance |
| CN113434376A (en) * | 2021-06-24 | 2021-09-24 | 山东浪潮科学研究院有限公司 | Web log analysis method and device based on NoSQL |
| CN113434376B (en) * | 2021-06-24 | 2023-04-11 | 山东浪潮科学研究院有限公司 | Web log analysis method and device based on NoSQL |
| CN115801353A (en) * | 2022-11-03 | 2023-03-14 | 智网安云(武汉)信息技术有限公司 | Linkage script processing method after real-time aggregation of safety event logs based on big data level |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108334556A (en) | A kind of method and system of analysis internet finance massive logs | |
| CN109522287B (en) | Monitoring method, system, equipment and medium for distributed file storage cluster | |
| CN107357804A (en) | The analysis system and method for internet finance massive logs | |
| US10116534B2 (en) | Systems and methods for WebSphere MQ performance metrics analysis | |
| US10747592B2 (en) | Router management by an event stream processing cluster manager | |
| CN111339175B (en) | Data processing method, device, electronic equipment and readable storage medium | |
| CN103295155A (en) | Security core service system monitoring method | |
| CN105122733A (en) | Queue monitoring and visualization | |
| CN111858251A (en) | Big data computing technology-based data security audit method and system | |
| CN111858274B (en) | Stability monitoring method for big data scoring system | |
| CN107656973A (en) | A kind of log audit subsystem applied to cloud auditing system | |
| US9037705B2 (en) | Routing of performance data to dependent calculators | |
| CN111130882A (en) | Monitoring system and method of network equipment | |
| Stefanov et al. | A review of supercomputer performance monitoring systems | |
| CN111240936A (en) | Data integrity checking method and equipment | |
| CN111796993B (en) | Data processing method and device, electronic equipment and computer readable storage medium | |
| CN111049846A (en) | Data processing method and device, electronic equipment and computer readable storage medium | |
| CN116506300A (en) | Website traffic data statistics method and system | |
| EP2770447B1 (en) | Data processing method, computational node and system | |
| CN115664992A (en) | Network operation data processing method and device, electronic equipment and medium | |
| CN114925283A (en) | Management method and system of push task, electronic device and medium | |
| Dzik et al. | Real-time AWS resources monitoring and analytics | |
| Fernández-Becerra et al. | Accountability as a service for robotics: Performance assessment of different accountability strategies for autonomous robots | |
| Chakraborty et al. | Observability | |
| CN113900898B (en) | Data processing system, equipment and medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| WD01 | Invention patent application deemed withdrawn after publication | ||
| WD01 | Invention patent application deemed withdrawn after publication |
Application publication date: 20180727 |