CN109634757A - A kind of collecting method of seismic industry big data processing - Google Patents
A kind of collecting method of seismic industry big data processing Download PDFInfo
- Publication number
- CN109634757A CN109634757A CN201811500930.8A CN201811500930A CN109634757A CN 109634757 A CN109634757 A CN 109634757A CN 201811500930 A CN201811500930 A CN 201811500930A CN 109634757 A CN109634757 A CN 109634757A
- Authority
- CN
- China
- Prior art keywords
- data
- acquisition
- collecting method
- database
- big data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Transfer Between Computers (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The collecting method that the present invention provides a kind of seismic industry big data processing includes the following steps: that data store step: all collected data being stored in Kafka cache cluster, and the message that data are stored in Kafka cache cluster is forwarded.The collecting method of seismic industry big data processing provided by the invention by docking with the data source in each operation system, provides the big data processing platform of seismic industry to the service such as data acquisition, unloading, processing.To solve the current problem encountered of data sharing by introducing big data technology.
Description
Technical field
The invention belongs to earthquake big data technical field, the data for being specifically related to a kind of seismic industry big data processing are adopted
Set method.
Background technique
Big data has become the Informatization Development direction of country, and China Seismological Network Center needs to rely on big data technology
New data sharing, data processing technique system are constructed, supports data sharing and existing earthquake using big data/cloud computing technology
Monitoring and prediction business forms the data sharing technology method across existing monitoring and prediction operation system, to build across multi-service system
The aggregation of data analytical technology of system is built based on big data/cloud computing technology technology platform blank.
China Seismological Network Center is the processing of national earthquake Observation Service and data sharing center, and main includes that country surveys shake
Platform net center, national Earthquake Precursory Station Network Center, seismic data sharing center, will also collect national Strong-motion Network, GNSS later
The observation data of land state network.
Therefore, by introducing big data technology, the current problem encountered of data sharing is solved.Meanwhile it exploring based on big
The new application model of earthquake monitoring and forecasting business under data/cloud computing technology, explores new experimental data correlation analysis.
Summary of the invention
Above-mentioned the deficiencies in the prior art are directed to, the purpose of the present invention is to provide a kind of processing of seismic industry big data
Collecting method.
To reach above-mentioned purpose, the present invention adopts the following technical scheme: a kind of data of seismic industry big data processing are adopted
Set method includes the following steps: that data store step: all collected data being stored in Kafka cache cluster, and will
The message that data are stored in Kafka cache cluster forwards.
It preferably, further include data collection steps before data store step: with the data source pair in each operation system
It connects, judges the acquisition adaptation mode of data, and data acquisition, the acquisition of the data are carried out based on determining acquisition adaptation mode
Adaptation mode includes: the acquisition of data-oriented source, data base-oriented acquisition and object oriented file acquisition.
Preferably, data-oriented source acquisition for can directly be docked with data source and collected data no longer
Variation is generated, the docking of shake real-time streams is surveyed or measuring instrument directly reports acquisition；System sends request data to streaming server
Account and password after being proved to be successful, return to address and the port numbers for receiving data, receive earthquake from the address of return and port numbers
Real-time streaming data is then forwarded to Kafka caching if recording the time point of receiving after receiving a complete bag data
Cluster.
Preferably, in the data base-oriented acquisition, the system access of database form, i.e., association is with relevant database
The open outer welding system of form, is associated and is monitored to the database under its system by adapter, to realize related service
The real-time or timing acquisition of information and date provides database by the way of through ETL tool in the realization of data acquisition
Adapter is completed: being connected each access service system database using the database adapter that ETL tool provides and is completed data
Acquisition and extraction, and can be realized in a manner of real-time, timing according to business need.
Preferably, in object oriented file acquisition: can only be handed over each operation system data by file or other transfer modes
Shift to big data platform.
Preferably, the data forwarding in data storage step includes full dose data forwarding and customization data forwarding；Institute
It states in full dose data forwarding, if user has subscribed certain data, subscribed data can be sent out in a manner of real-time streams
Give subscription end；In the customization data forwarding, the hobby from full dose data according to user filters out the number of user's care
According to being then forwarded to user.
Compared to the prior art, technical solution provided by the invention has the following beneficial effects:
The collecting method of seismic industry big data processing provided by the invention handles the big data of seismic industry flat
Platform provides the service such as data acquisition, unloading, processing by docking with the data source in each operation system.To big by introducing
Data technique solves the current problem encountered of data sharing.
Detailed description of the invention
The drawings described herein are used to provide a further understanding of the present invention, constitutes a part of the invention, this hair
Bright illustrative embodiments and their description are used to explain the present invention, and are not constituted improper limitations of the present invention.In the accompanying drawings:
Fig. 1 is the mistake of data-oriented source acquisition in the collecting method of seismic industry big data processing provided by the invention
Journey schematic diagram；
Fig. 2 is the process signal of data forwarding in the collecting method of seismic industry big data processing provided by the invention
Figure.
Specific embodiment
In order to be clearer and more clear technical problems, technical solutions and advantages to be solved, tie below
Drawings and examples are closed, the present invention will be described in further detail.It should be appreciated that specific embodiment described herein is only used
To explain the present invention, it is not intended to limit the present invention.
In claims of the present invention, specification and above-mentioned attached drawing, unless otherwise specifically limited, such as using term " the
One ", " second " or " third " etc. are provided to distinguish different objects, be not use to describe a particular order.
In claims of the present invention, specification and above-mentioned attached drawing, such as using term " includes ", " having " and they
Deformation, it is intended that " including but not limited to ".
A kind of collecting method of seismic industry big data processing includes the following steps:
Data collection steps: docking with the data source in each operation system, judges the acquisition adaptation mode of data, and is based on
Determining acquisition adaptation mode carries out data acquisition, the acquisition adaptation mode of the data include: the acquisition of data-oriented source, towards
Database acquisition and object oriented file acquisition；
Data store step: all collected data being stored in Kafka cache cluster, and data are stored in
Message in Kafka cache cluster forwards.
In data collection steps, for acquiring towards data source, the data-oriented source acquisition is used for can be direct
It is docked with data source and collected data no longer generates variation, surveyed the docking of shake real-time streams or measuring instrument is directly reported and adopted
Collection；
System sends the account and password of request data to streaming server, after being proved to be successful, returns to the address for receiving data
And port numbers, receive earthquake real-time streaming data from the address of return and port numbers, if after receiving a complete bag data,
The time point received is recorded, Kafka cache cluster is then forwarded to.
Specifically, in the collection process of data-oriented source, as shown in Figure 1, LISS flow data is as follows the step of docking process:
1, creation socket requests to obtain LISS flow data address
The socket connection for creating real-time streams first, sends the user name password request of docking real-time streams, the life of request
It enables: " user LISS flows Yong Huming r npass LISS Liu Mima r npasv rt n ", if verifying does not pass through, LISS stream
Server returns to error message.LISS streaming server returns to a string of character strings by socket connection if the verification passes:
227Real Time Data Port Entering Passive Mode (ip, port), by processing extract the inside ip and
Port, this ip and port are exactly the address for receiving server LISS flow data.
2, creation receives socket, sends data reception command
A received socket of data is created with the ip and port parsed, then sends data request command:
Retr seismic* r n. then will to send in the socket of data receiver be real-time to LISS streaming server
Flow data.
3, data transfer integrity is handled
Since there are this unstable factors, such as network to disconnect for network transmission, data packet splicing.
A) socket connection uses TCP connection, and when there is the case where network middle-end, data collection terminal needs to re-create number
It is connected according to request, sends LISS user name and connected with LISS password request data with data receiver and send request of data name.
B) the data fixed size of LISS stream is 512Byte.So data receiver is to fix the size of data of 512 bytes
For a data packet.
4, data reconnection mechanism
When TCP connection disconnects, system automatically detects disconnection, and according to every 30 seconds trial reconnection LISS streaming servers into
Row request data.
5, data are sent to message trunking
Complete 512 byte data after system will be handled well stamps the label of time and (records what when this real-time streams are
Between receive), be stored in inside an object, object serialized, message trunking is then forwarded to.
For data base-oriented acquisition, in data base-oriented acquisition, the system access of database form,
That is association outer welding system open in the form of relevant database, the database under its system is associated by adapter and
Monitoring, to realize the real-time or timing acquisition of related service information and date, in the realization of data acquisition, using passing through ETL
The mode of tool provides database adapter and is completed: connecting each access industry using the database adapter that ETL tool provides
The acquisition and extraction for system database completion data of being engaged in, and can be realized in a manner of real-time, timing according to business need.
For object oriented file acquisition, in object oriented file acquisition: can only be incited somebody to action by file or other transfer modes
Each operation system data Jiao Huanzhi big data platform.
In addition, data forwarding includes full dose data forwarding and customization data forwarding in data storage step.Moreover, such as
Shown in Fig. 2, data storage is first to judge whether the data obtained are complete correct after obtaining data in Kafka cluster
Earthquake real-time streaming data, if not with regard to record log and abandon this record, then judge whether if it is complete data
For 1970 invalid datas, need to judge whether to meet the requirement of forwarding if it is valid data, it is just direct if it is full dose forwarding
Forwarding needs to judge whether the data needs to forward if it is customization forwarding, and meet forwarding condition is forwarded to designated position.
It, can be subscribed data with reality if user has subscribed certain data in the full dose data forwarding
When the mode that flows be sent to subscription end；Such as: party A-subscriber has subscribed all data being collected into Kafka cluster, when Kafka collection
Group can all be transmitted to party A-subscriber after receiving data in a manner of real-time streams.User receives do-it-yourself data processing after data.
If what multi-user subscribed to is same part data, only the initial time of different user access evidence is different.
In the customization data forwarding, the hobby from full dose data according to user filters out the data of user's care,
It is then forwarded to user.Such as party A-subscriber has customized the real-time streaming data of certain several station, data processing module can customize user
Data screened and then be stored in Kafka cluster, be then forwarded to user A, data reusing problem, if user B and
The data of user A customization are equally, then to house a common data in Kafka cluster, store if different
More parts of data.
Specifically, for customizing data forwarding, specific step is as follows for the customization data forwarding:
1, customization forwarding needs to create the table which station each user of storage has customized first.Including number, ip
Address, user, password, station id (multiple use, segmentation), the application time ratifies the time, state, date created, founder, more
New time, regenerator, mark delete mark, receive identification field.
2, when user applies customizing the data of certain stations, an application record can be increased newly inside this table, wait Shen
After please ratifying, identification number takes the data of customization to user based on the received.
3, when background process customization data, the application inside custom table after searching and managing person's approval, traversal can first be gone
The record of all customization approvals forms one group of key-value pair data, and key assignments is the ID of the station, and numerical value is to receive mark, if there is
Multiple just cumulative additions, in this way when system receives certain data, go in caching to inquire this station data whether
There is user's customization, if not then do not handle, if there is just taking out numerical value, data is sent to each and is received in mark.This
Sample user respectively go receive identify in access according to when can.
4, it needs to determine because of whether comparison data in the buffer needs to be sent to the data in reception mark, in caching
When go to update.A more new thread is created, the custom table data in inquiry data is then periodically gone, then updates in caching
It goes.
The preferred embodiment of the present invention has shown and described in above description, as previously described, it should be understood that the present invention is not office
Be limited to form disclosed herein, should not be regarded as an exclusion of other examples, and can be used for various other combinations, modification and
Environment, and can be changed within that scope of the inventive concept describe herein by the above teachings or related fields of technology or knowledge
It is dynamic.And changes and modifications made by those skilled in the art do not depart from the spirit and scope of the present invention, then it all should be appended by the present invention
In scope of protection of the claims.
Claims (6)
1. a kind of collecting method of seismic industry big data processing, characterized by the following steps:
Data store step: all collected data being stored in Kafka cache cluster, and data are stored in Kafka
Message in cache cluster forwards.
2. the collecting method of seismic industry big data processing according to claim 1, it is characterised in that: deposited in data
Strideing rapid further includes before data collection steps: docking with the data source in each operation system, judges the acquisition adaptation side of data
Formula, and data acquisition is carried out based on determining acquisition adaptation mode, the acquisition adaptation mode of the data includes: data-oriented source
Acquisition, data base-oriented acquisition and object oriented file acquisition.
3. the collecting method of seismic industry big data according to claim 2 processing, it is characterised in that: it is described towards
Data source acquisition is used to directly to dock with data source and collected data no longer generate variation, surveys shake real-time streams docking
Or measuring instrument directly reports acquisition；
System sends the account and password of request data to streaming server, after being proved to be successful, returns to the address and end for receiving data
Slogan receives earthquake real-time streaming data from the address of return and port numbers, if after receiving a complete bag data, record
The time point of receiving is then forwarded to Kafka cache cluster.
4. the collecting method of seismic industry big data according to claim 2 processing, it is characterised in that: it is described towards
In database acquisition,
The system access of database form, i.e. association outer welding system open in the form of relevant database, pass through adapter pair
Database under its system is associated and monitors, to realize the real-time or timing acquisition of related service information and date, in number
According in the realization of acquisition, database adapter is provided by the way of through ETL tool and is completed: being provided using ETL tool
Database adapter connect acquisition and extraction that each access service system database completes data, and can according to business need,
It is realized in a manner of real-time, timing.
5. the collecting method of seismic industry big data processing according to claim 2, it is characterised in that: towards text
In part acquisition:
It can only be by file or other transfer modes by each operation system data Jiao Huanzhi big data platform.
6. the collecting method of seismic industry big data processing according to claim 1, it is characterised in that: deposited in data
The data forwarding in suddenly of strideing includes full dose data forwarding and customization data forwarding；
It, can be subscribed data with real-time streams if user has subscribed certain data in the full dose data forwarding
Mode be sent to subscription end；
In the customization data forwarding, the hobby from full dose data according to user filters out the data of user's care, then
It is transmitted to user.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811500930.8A CN109634757B (en) | 2018-12-10 | 2018-12-10 | Data acquisition method for seismic industry big data processing |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811500930.8A CN109634757B (en) | 2018-12-10 | 2018-12-10 | Data acquisition method for seismic industry big data processing |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109634757A true CN109634757A (en) | 2019-04-16 |
| CN109634757B CN109634757B (en) | 2022-10-11 |
Family
ID=66072418
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811500930.8A Active CN109634757B (en) | 2018-12-10 | 2018-12-10 | Data acquisition method for seismic industry big data processing |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109634757B (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110262999A (en) * | 2019-06-03 | 2019-09-20 | 北京一览群智数据科技有限责任公司 | A kind of circulation of automated data and data processing method, shared file server |
| CN113377841A (en) * | 2021-06-21 | 2021-09-10 | 国网宁夏电力有限公司电力科学研究院 | Big data-based energy load prediction system |
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108681489A (en) * | 2018-05-25 | 2018-10-19 | 西安交通大学 | It is a kind of it is super calculate environment under mass data in real time acquisition and processing method |
-
2018
- 2018-12-10 CN CN201811500930.8A patent/CN109634757B/en active Active
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108681489A (en) * | 2018-05-25 | 2018-10-19 | 西安交通大学 | It is a kind of it is super calculate environment under mass data in real time acquisition and processing method |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110262999A (en) * | 2019-06-03 | 2019-09-20 | 北京一览群智数据科技有限责任公司 | A kind of circulation of automated data and data processing method, shared file server |
| CN113377841A (en) * | 2021-06-21 | 2021-09-10 | 国网宁夏电力有限公司电力科学研究院 | Big data-based energy load prediction system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN109634757B (en) | 2022-10-11 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108667725A (en) | A kind of industrial AnyRouter and implementation method based on a variety of accesses and edge calculations | |
| CN107844421A (en) | Interface test method, device, computer equipment and storage medium | |
| CN102468981B (en) | Systems and methods for business network management discovery and consolidation | |
| CN107818127A (en) | A kind of querying method and system for multi-source data | |
| CN102123044B (en) | Detection device and method of network topology consistency based on topology discovery technology | |
| CN108959337A (en) | Big data acquisition methods, device, equipment and storage medium | |
| CN107147535A (en) | A kind of distributed network measurement data statistical analysis technique | |
| CN108200111A (en) | Resource allocation information update method, device and resource interface equipment | |
| CN106294826A (en) | A kind of company-data Query method in real time and system | |
| CN108833264A (en) | Data acquisition management system, method and application based on wechat small routine | |
| CN109634757A (en) | A kind of collecting method of seismic industry big data processing | |
| CN104486116A (en) | Multidimensional query method and multidimensional query system of flow data | |
| CN100449533C (en) | Providing information for mobile user | |
| CN109815286A (en) | A kind of adaptive telemetry of launch vehicle system and its implementation | |
| Prelipcean et al. | Workshop Synthesis: New developments in travel diary collection systems based on smartphones and GPS receivers | |
| CN110442590A (en) | It is a kind of for provide examine detection service system and method | |
| SE9801525L (en) | Procedure and device in computer networks | |
| CN104750860B (en) | A kind of date storage method of uncertain data | |
| CN108563706A (en) | A kind of collection big data intelligent service system and its operation method | |
| CN108764832A (en) | Municipal administration and public sentiment demand approaches to IM, system, device and equipment | |
| CN101262481A (en) | A remote service recognition system and method for computer network | |
| CN114443750A (en) | Resource data storage and query method and device | |
| CN102891781B (en) | Network shares detection system and network shares detection method | |
| CN110413496A (en) | A method of realizing that the operation data modularization of electronics license is collected | |
| CN110544182A (en) | Power distribution communication network fusion control method and system based on machine learning technology |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |