CN106850763A - Data distribution formula is received and analysis method and system - Google Patents
Data distribution formula is received and analysis method and system Download PDFInfo
- Publication number
- CN106850763A CN106850763A CN201710005865.0A CN201710005865A CN106850763A CN 106850763 A CN106850763 A CN 106850763A CN 201710005865 A CN201710005865 A CN 201710005865A CN 106850763 A CN106850763 A CN 106850763A
- Authority
- CN
- China
- Prior art keywords
- data
- distributed
- analysis
- received
- distribution formula
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/06—Protocols specially adapted for file transfer, e.g. file transfer protocol [FTP]
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
Data distribution formula of the present invention is received and analysis method, including：Step 1, distributed collection data；Step 2, download collect data, and to download data carry out distributed analysis；Step 3, stores by the data of distributed analysis.Data distribution formula reception of the present invention and analysis method and system provide multiple spot and collect data and upload to cloud storage, the end for collecting data can be disposed with magnanimity, which enhance storage and the analyze speed of data, storage for relative complex data becomes simple, easy with analysis, and storing can each carry out linear expansion and improve as needed with analysis so that analyzing system performance is optimal.
Description
Technical field
The present invention relates to a kind of data receiver and analysis method, especially a kind of data distribution formula receive and analysis method and
System.
Background technology
With the appearance and popularization of computer, the information of magnanimity is present in network everywhere in digitized form, science data,
Medical data, consensus data, finance data etc., these data have also attracted substantial amounts of network attack.And with net
Network attacks more and more, becomes increasingly complex, and related security event data species is various, and data structure is also and its complexity.
At present, generally all it is to copy a system to for the reception of data, then data is analyzed, without using big
The facility of data storage, if log server is a lot, it will increase many workloads, can also take a lot of time.
The content of the invention
For defect of the prior art, operating efficiency is improved present invention aim at there is provided a kind of workload that reduces
Data distribution formula is received and analysis method and system.
In order to solve the above technical problems, data distribution formula of the present invention is received and analysis method, including：Step 1, distribution is received
Collection data；Step 2, download collect data, and to download data carry out distributed analysis；Step 3, storage is by distributed
The data of analysis.
Preferably, step 1 comprises the following steps：Step 1.1, distributed collection data are forming journal file；Step
1.2, journal file is uploaded.
Preferably, in step 1.1, by open source software RTKLIB distributed collections data forming journal file.
Preferably, step 2 comprises the following steps：Step 2.1, timing inquiry upload journal file, when inquire with advance
If during the journal file that condition matches, downloading the journal file；Step 2.2, the journal file to downloading carries out distributed place
Reason.
Preferably, it is pre-conditioned including the time and type in step 2.1.
Preferably, in step 2.2, distributed treatment includes resolving time, fixed rate or precision.
Preferably, journal file includes time, state, longitude and latitude, height and precision.
Preferably, also shown by the data of distributed analysis including step 4.
Data system, including：Cloud storage service module, cloud storage service module is distributed from multiple log servers to be received
Collection data；Distributed analysis module, distributed analysis module downloading data from cloud storage service module, and the data to downloading
Carry out distributed analysis；Memory module, memory module receives the number by distributed analysis uploaded from distributed analysis module
According to；Display module, display module is shown by the data of distributed analysis.
Preferably, cloud storage service module is OSS；Memory module is cloud database.
Data distribution formula reception of the present invention and analysis method and system provide multiple spot and collect data and upload to cloud storage, receive
The end for collecting data can be disposed with magnanimity, storage and the analyze speed of data be which enhanced, for depositing for relative complex data
Storage becomes simple, easy with analysis, and storage can each carry out linear expansion and improve as needed with analysis so that divide
Analysis system performance is optimal.
Brief description of the drawings
The detailed description made to non-limiting example with reference to the following drawings by reading, further feature mesh of the invention
And advantage will become more apparent upon.
Fig. 1 is that data distribution formula of the present invention is received and analysis method flow chart；
Fig. 2 is data system schematic diagram of the present invention.
Specific embodiment
With reference to specific embodiment, the present invention is described in detail.Following examples will be helpful to the technology of this area
Personnel further understand the present invention, but the invention is not limited in any way.It should be pointed out that to the ordinary skill of this area
For personnel, without departing from the inventive concept of the premise, some changes and improvements can also be made.These belong to the present invention
Protection domain.
The present invention provides multiple spot and collects data and upload to cloud storage, and the end for collecting data can be disposed with magnanimity, analyzes number
According to when, from cloud storage downloading data and be analyzed, the data Cun Chudao distributed data-storage systems that analysis collects.
As shown in Figure 1 and Figure 2, data distribution formula of the present invention is received and analysis method, including：Step 1, distributed collection number
According to；Step 2, download collect data, and to download data carry out distributed analysis；Step 3, stores by distributed analysis
Data.
Step 1 comprises the following steps：It is literary that step 1.1, distributed collection data and the data summarization that will be collected form daily record
Part；Step 1.2, journal file is uploaded.
In step 1.1, by open source software RTKLIB distributed collections data and will collect data summarization formed daily record
File.
Step 2 comprises the following steps：Step 2.1, the journal file that timing inquiry is uploaded, when inquiring and pre-conditioned phase
During identical journal file, the journal file is downloaded；Step 2.2, the journal file to downloading carries out distributed treatment.
It is pre-conditioned including the time and type in step 2.1.
In step 2.2, distributed treatment includes resolving time, fixed rate or precision.
Journal file includes time, state, longitude and latitude, height and precision.
Also include step 4, show by the data of distributed analysis.
Data system, including：Cloud storage service module, cloud storage service module is distributed from multiple log servers to be received
Collection data；Distributed analysis module, distributed analysis module downloading data from cloud storage service module, and the data to downloading
Carry out distributed analysis；Memory module, memory module receives the number by distributed analysis uploaded from distributed analysis module
According to；Display module, display module is shown by the data of distributed analysis.
Cloud storage service module is OSS；Memory module is cloud database.
The present invention is using cloud storage and the advantage of big data storage platform, in terms of the collection of data, each log services
Used as a point, the automatic JFS that uploads uploads to cloud storage to device as auxiliary timing.Analysis system is timed to cloud and deposits
Storage inquiry scheduled date and the file of type, it is locally downloading and carry out analytic solution calculation.The type with data is collected as needed
The data for collecting are saved in big data storage platform.Front end represents system and arrives the reading of big data storage platform and demonstrating data again.
Below the preferred embodiment to the invention is illustrated, but the invention be not limited to it is described
Embodiment, those of ordinary skill in the art can also make a variety of equivalent on the premise of without prejudice to the invention spirit
Modification or replacement, these equivalent modifications or replacement are all contained in the application claim limited range.
Claims (10)
1. data distribution formula is received and analysis method, it is characterised in that comprised the following steps：
Step 1, distributed collection data；
Step 2, download collect data, and to download data carry out distributed analysis；
Step 3, stores by the data of distributed analysis.
2. data distribution formula according to claim 1 is received and analysis method, it is characterised in that step 1 includes following step
Suddenly：
Step 1.1, distributed collection data are forming journal file；
Step 1.2, journal file is uploaded.
3. data distribution formula according to claim 2 is received and analysis method, it is characterised in that in step 1.1, by opening
Source software RTKLIB distributed collections data are forming journal file.
4. data distribution formula according to claim 1 is received and analysis method, it is characterised in that step 2 includes following step
Suddenly：
Step 2.1, the journal file that timing inquiry is uploaded, when the journal file matched with pre-conditioned is inquired, is downloaded
The journal file；
Step 2.2, the journal file to downloading carries out distributed treatment.
5. data distribution formula according to claim 4 is received and analysis method, it is characterised in that in step 2.1, presets bar
Part includes time and type.
6. data distribution formula according to claim 4 is received and analysis method, it is characterised in that in step 2.2, distributed
Treatment includes resolving time, fixed rate or precision.
7. the data distribution formula according to Claims 2 or 3 or 4 is received and analysis method, it is characterised in that journal file bag
Include time, state, longitude and latitude, height and precision.
8. data distribution formula according to claim 1 is received and analysis method, it is characterised in that also including step 4, display
By the data of distributed analysis.
9. data system, it is characterised in that including：
Cloud storage service module, cloud storage service module distributed collection data from multiple log servers；
Distributed analysis module, distributed analysis module downloading data from cloud storage service module, and to download data enter
Row distributed analysis；
Memory module, memory module receives the data by distributed analysis uploaded from distributed analysis module；
Display module, display module is shown by the data of distributed analysis.
10. data system according to claim 9, it is characterised in that cloud storage service module is OSS；Deposit
Storage module is cloud database.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710005865.0A CN106850763A (en) | 2017-01-04 | 2017-01-04 | Data distribution formula is received and analysis method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710005865.0A CN106850763A (en) | 2017-01-04 | 2017-01-04 | Data distribution formula is received and analysis method and system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN106850763A true CN106850763A (en) | 2017-06-13 |
Family
ID=59117804
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710005865.0A Pending CN106850763A (en) | 2017-01-04 | 2017-01-04 | Data distribution formula is received and analysis method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106850763A (en) |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20070143842A1 (en) * | 2005-12-15 | 2007-06-21 | Turner Alan K | Method and system for acquisition and centralized storage of event logs from disparate systems |
| CN105677842A (en) * | 2016-01-05 | 2016-06-15 | 北京汇商融通信息技术有限公司 | Log analysis system based on Hadoop big data processing technique |
| CN105893628A (en) * | 2016-05-17 | 2016-08-24 | 中国农业银行股份有限公司 | Real-time data collection system and method |
-
2017
- 2017-01-04 CN CN201710005865.0A patent/CN106850763A/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20070143842A1 (en) * | 2005-12-15 | 2007-06-21 | Turner Alan K | Method and system for acquisition and centralized storage of event logs from disparate systems |
| CN105677842A (en) * | 2016-01-05 | 2016-06-15 | 北京汇商融通信息技术有限公司 | Log analysis system based on Hadoop big data processing technique |
| CN105893628A (en) * | 2016-05-17 | 2016-08-24 | 中国农业银行股份有限公司 | Real-time data collection system and method |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN105490854B (en) | Real-time logs collection method, system and application server cluster | |
| Ihm et al. | Towards understanding modern web traffic | |
| CN106933724B (en) | Distributed information tracking system, information processing method and device | |
| CN104077402B (en) | Data processing method and data handling system | |
| CN107908690A (en) | A kind of data processing method based on big data OA operation analysis | |
| CN101192227B (en) | Log file analytical method and system based on distributed type computing network | |
| CN104144142B (en) | A kind of Web bug excavation methods and system | |
| CN103927370B (en) | Network information batch acquisition method of combined text and picture information | |
| CN104065532B (en) | A kind of non-recorded website search method and system based on multichannel data access way | |
| CN102164186B (en) | Method and system for realizing cloud search service | |
| CN103178982A (en) | Method and device for analyzing log | |
| CN104572976B (en) | Website data update method and system | |
| CN105634845A (en) | Method and system for carrying out multi-dimensional statistic analysis on large number of DNS journals | |
| CN101409690A (en) | Method and system for obtaining internet user behaviors | |
| CN105718590A (en) | Multi-tenant oriented SaaS public opinion monitoring system and method | |
| CN102833233B (en) | Method and device for recognizing web pages | |
| CN107729219A (en) | Resource monitoring method, device and terminal based on super fusion storage system | |
| CN104636232A (en) | Performance monitoring device and method of distributed service system | |
| CN101282342A (en) | Method and system for fetching network contents | |
| CN111459986A (en) | Data computing system and method | |
| CN105930502B (en) | System, client and method for collecting data | |
| US8504656B2 (en) | Systems and methods for collecting and storing network traffic data | |
| US20180293089A1 (en) | Method and device for adding indicative icon in interactive application | |
| CN106850763A (en) | Data distribution formula is received and analysis method and system | |
| CN105550265A (en) | Quasi-real-time user log collecting and processing method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20170613 |