CN110377439A - A kind of system of real time data acquisition push - Google Patents
A kind of system of real time data acquisition push Download PDFInfo
- Publication number
- CN110377439A CN110377439A CN201910665423.8A CN201910665423A CN110377439A CN 110377439 A CN110377439 A CN 110377439A CN 201910665423 A CN201910665423 A CN 201910665423A CN 110377439 A CN110377439 A CN 110377439A
- Authority
- CN
- China
- Prior art keywords
- data
- module
- real time
- web interface
- websocket
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000012545 processing Methods 0.000 claims abstract description 21
- 230000027455 binding Effects 0.000 claims abstract description 8
- 238000009739 binding Methods 0.000 claims abstract description 8
- 230000000694 effects Effects 0.000 claims description 2
- 238000005516 engineering process Methods 0.000 description 4
- 238000004891 communication Methods 0.000 description 3
- 238000010586 diagram Methods 0.000 description 3
- 238000013079 data visualisation Methods 0.000 description 2
- 238000000034 method Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000013480 data collection Methods 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000001747 exhibiting effect Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 238000010025 steaming Methods 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
- 238000012795 verification Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24568—Data stream processing; Continuous queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/252—Integrating or interfacing systems involving database management systems between a Database Management System and a front-end application
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/284—Relational databases
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/544—Buffers; Shared memory; Pipes
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/547—Remote procedure calls [RPC]; Web services
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Data Mining & Analysis (AREA)
- Computational Linguistics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Information Transfer Between Computers (AREA)
Abstract
The invention discloses a kind of systems of real time data acquisition push characterized by comprising kafka module, data processing module, websocket module, data memory module and web interface module, kafka module real-time release data to be subscribed to；The data processing module subscribes to data from the kafka module, and the websocket module and the data memory module are sent to after being verified；The data memory module stores the data received from the data processing module；The websocket module and the web interface module communicate to connect, by the data-pushing received from the data processing module to the web interface module；The web interface module carries out data double-way binding, is shown to received data.To realize real time data acquisition real-time exhibition.
Description
Technical field
The present invention relates to the systems that data visualization technique field more particularly to real time data acquisition push.
Background technique
When traditional data is shown, interface of page initialization requests, it is only necessary to be counted from the data of database
It shows, this, which means that, wants to see data, can only refresh a page every time.Until a kind of ajax (creation interaction
The web development technologies of formula web application) technology appearance, it is no longer necessary to refresh page, timing request interface from reach data reality
Shi Xiaoguo.But with the rise of big data, in terms of data exhibiting, to data to acquire the demand that shows in real time higher and higher.
Some units and enterprise, which need to reach the operation data on the same day, to be showed state in real time or shows Various types of data index in large-size screen monitors.This
Require acquiring and show in real time for energy data, ajax technology can no longer meet demand, especially in terms of data visualization.
Summary of the invention
The purpose of the present invention is to provide a kind of system of real time data acquisition push, realize that real time data acquisition is opened up in real time
Show.
Realizing the technical solution of above-mentioned purpose is:
A kind of system of real time data acquisition push, comprising: a kafka (distribution of Apache Software Foundation open source
Formula steaming media platform) module, data processing module, websocket be (a kind of that full-duplex communication is carried out in the single TCP connection
Agreement) module, data memory module and web (WWW) interface module, wherein
Kafka module real-time release data to be subscribed to；
The data processing module subscribes to data from the kafka module, is sent to after being verified described
Websocket module and the data memory module；
The data memory module stores the data received from the data processing module；
The websocket module and the web interface module communicate to connect, will be received from the data processing module
Data-pushing is to the web interface module；
The web interface module carries out data double-way binding, is shown to received data.
Preferably, the web interface module using vue.js (a set of for constructing the gradual frame of user interface) into
The binding of row data double-way.
Preferably, it when the web interface module does not establish connection with the websocket module, is stored from the data
Module carries out acquisition data.
Preferably, the data memory module uses a kind of mysql (Relational DBMS) database.
Preferably, when the data processing module verifies data, each data field for first checking for data is
It is no complete, if lacking any one, determine this data invalid；If complete, according to the respective default rule of each data field, to each
A data field is verified one by one, if after passing through, decision verification is qualified.
Preferably, respectively default rule includes: character length limitation, spcial character limitation to each data field
It is limited with character types.
The beneficial effects of the present invention are: the present invention using kafka three kinds of characteristics (1. can allow you issue and subscribe to streaming
Note.2. the record of streaming can be stored, and there is preferable fault-tolerance.3. can just be located when streaming is recorded and generated
Reason) carry out real-time data collection.Meanwhile pushed data in real time is carried out to web page using websocket, it is real-time to reach data
Acquire the effect of real-time exhibition.Also, the present invention, come storing data, is read from database using mysql when page initialization
Access evidence, the energy quick response in page initialization, improves user experience.
Detailed description of the invention
Fig. 1 is that Ei is reached figure by the system of real time data acquisition push of the invention；
Fig. 2 is that the producer gives out information to kafka, schematic diagram of the consumer from kafka subscription message in the present invention；
Fig. 3 is the schematic diagram of data processing in the present invention；
Fig. 4 is the schematic diagram that websocket module and web interface module carry out communication bindings in the present invention.
Specific embodiment
The present invention will be further described with reference to the accompanying drawings.
Referring to Fig. 1, the system that real time data acquisition of the invention pushes, comprising: kafka module 1, data processing module
2, websocket module 3, data memory module 4 and web interface module 5.
1 real-time release of kafka module data to be subscribed to.Kafka module 1 is accomplished by the following way:
Kafka, starting kakfa service are installed on a server, while creating a theme is " bigdata ",
Producers (producer) by data publication to bigdata theme, one or more Consumer (consumer) from
Bigdata theme subscribes to data.Such as: issued into bigdata theme json data " a ": 1, " b ": 2, " c ":
3,"d":4,"e":5}.As shown in Figure 2.
Data processing module 2 subscribes to data from kafka module 1, and 3 He of websocket module is sent to after being verified
Data memory module 4.As shown in figure 3, data processing module 2, websocket module 3, data storage is accomplished by the following way
Module 4 and web interface module 5:
Mysql is installed on a Nginx server.By writing php (HyperText Preprocessor) script, such as: claim
It is named as kafkaConsumer.php, subscribes to the data of bigdata theme in real time as kafka client, to the data of subscription
It is verified, such as subscribes to hotel's data, provided by rule, data field must include hotel name, hotel address name
Claim, room number moves in people's name, and passport NO. is moved in the time, administrative division number, the check-out time.When the data of subscription lack must
The data field filled out any one when, determine that this data is invalid data, when the data field that must be filled out all meets, according to each
From default rule (character length limitation, spcial character limitation and character types limitation etc.), each field is verified one by one, example
Such as: moving in people's name length is 4-16 character, does not allow spcial character such as@/%/$/&/* occur, if type of credential is two
Generation ID, passport NO. should meet national standard establishment.After the data field of this data presses rule by verifying one by one,
Then data are carried out being pushed to websocket module 3 and data memory module 4.To realize data processing module 2.Such as:
The producer has issued a json data, therefore this json data can be obtained in kafkaConsumer.php script, to json
Data carry out verify data after being converted to array, if only needing the data of c and d, store the number of c and d in data memory module 4
According to, while the data-pushing of c and d to websocket module 3.
A database is established in mysql database with a table, for storing the data subscribed to.To realize data
Memory module 4.Data memory module 4 stores the data received from data processing module 2；
Php script is write, such as: name nominating socket.php, as websocket module 3.As shown in figure 4,
Websocket module 3 and web interface module 5 communicate to connect, by the data-pushing received from data processing module 2 to web interface
Module 5.In Fig. 3, the data for having c and d of websocket module 3 are pushed to, websocket module 3 is immediately to the number of c and d
According to being pushed to web interface module 5.
Web interface module 5 using vue.js carry out data double-way binding, such as initial bind c, d value be 0, with
After websocket module 3 establishes communication connection, received data are shown.In websocket module 3, push comes
C and the data of d be 3,4 respectively.The data that real-time changing push comes as long as the place of binding c and d so on the page.
When web interface module 5 does not establish connection with websocket module 3, for example, when the page initialization when, from data memory module 4 into
Row obtains data.
Above embodiments are used for illustrative purposes only, rather than limitation of the present invention, the technology people in relation to technical field
Member, without departing from the spirit and scope of the present invention, can also make various transformation or modification, therefore all equivalent
Technical solution also should belong to scope of the invention, should be limited by each claim.
Claims (6)
1. a kind of system of real time data acquisition push characterized by comprising kafka module, data processing module,
Websocket module, data memory module and web interface module, wherein
Kafka module real-time release data to be subscribed to；
The data processing module subscribes to data from the kafka module, and the websocket mould is sent to after being verified
Block and the data memory module；
The data memory module stores the data received from the data processing module；
The websocket module and the web interface module communicate to connect, by the data received from the data processing module
It is pushed to the web interface module；
The web interface module carries out data double-way binding, is shown to received data.
2. the system of real time data acquisition push according to claim 1, which is characterized in that the web interface module is adopted
Data double-way binding is carried out with vue.js.
3. the system of real time data acquisition push according to claim 1, which is characterized in that the web interface module is not
When establishing connection with the websocket module, acquisition data are carried out from the data memory module.
4. the system of real time data acquisition push according to claim 1, which is characterized in that the data memory module is adopted
With mysql database.
5. the system of real time data acquisition push according to claim 1, which is characterized in that the data processing module pair
When data are verified, whether each data field for first checking for data complete, if lacking any one, determine this data without
Effect；If complete, according to the respective default rule of each data field, each data field is verified one by one, if determining after passing through
Verifying is qualified.
6. the system of real time data acquisition push according to claim 5, which is characterized in that each data field
Respective default rule includes: character length limitation, spcial character limitation and character types limitation.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910665423.8A CN110377439A (en) | 2019-07-23 | 2019-07-23 | A kind of system of real time data acquisition push |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910665423.8A CN110377439A (en) | 2019-07-23 | 2019-07-23 | A kind of system of real time data acquisition push |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110377439A true CN110377439A (en) | 2019-10-25 |
Family
ID=68255043
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910665423.8A Pending CN110377439A (en) | 2019-07-23 | 2019-07-23 | A kind of system of real time data acquisition push |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110377439A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111049846A (en) * | 2019-12-20 | 2020-04-21 | 北京明略软件系统有限公司 | Data processing method and device, electronic equipment and computer readable storage medium |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20130185366A1 (en) * | 2012-01-05 | 2013-07-18 | Joby Joy | System and method for managing user generated content |
| CN108040092A (en) * | 2017-11-28 | 2018-05-15 | 特斯联（北京）科技有限公司 | A kind of management of Internet of Things big data and application platform towards garden |
| US20180167476A1 (en) * | 2016-12-12 | 2018-06-14 | Sap Se | Meta broker for publish-subscribe-based messaging |
| CN108390881A (en) * | 2018-02-27 | 2018-08-10 | 北京焦点新干线信息技术有限公司 | A kind of distribution high concurrent real-time messages method for pushing and system |
| CN109271412A (en) * | 2018-09-28 | 2019-01-25 | 中国-东盟信息港股份有限公司 | The real-time streaming data processing method and system of smart city |
-
2019
- 2019-07-23 CN CN201910665423.8A patent/CN110377439A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20130185366A1 (en) * | 2012-01-05 | 2013-07-18 | Joby Joy | System and method for managing user generated content |
| US20180167476A1 (en) * | 2016-12-12 | 2018-06-14 | Sap Se | Meta broker for publish-subscribe-based messaging |
| CN108040092A (en) * | 2017-11-28 | 2018-05-15 | 特斯联（北京）科技有限公司 | A kind of management of Internet of Things big data and application platform towards garden |
| CN108390881A (en) * | 2018-02-27 | 2018-08-10 | 北京焦点新干线信息技术有限公司 | A kind of distribution high concurrent real-time messages method for pushing and system |
| CN109271412A (en) * | 2018-09-28 | 2019-01-25 | 中国-东盟信息港股份有限公司 | The real-time streaming data processing method and system of smart city |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111049846A (en) * | 2019-12-20 | 2020-04-21 | 北京明略软件系统有限公司 | Data processing method and device, electronic equipment and computer readable storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106250987B (en) | A kind of machine learning method, device and big data platform | |
| CN109582722A (en) | Public security resource data service system | |
| CN104462226B (en) | A kind of network based on cloud sings the construction method of platform | |
| CN105635716B (en) | Aggregated media method based on augmented reality | |
| CN105825120B (en) | The method and system of interactive authentication between a kind of computer and mobile terminal | |
| CN109981711A (en) | Document dynamic play method, apparatus, system and computer readable storage medium | |
| CN104408984A (en) | Intelligent teaching system comprising multiple teaching terminals | |
| CN101188620A (en) | Method for creating Internet virtual reception hall and realizing synchronous and asynchronous exchange by using flash plug-in technology | |
| CN108600777A (en) | A kind of stream media playback server, client | |
| CN103327278A (en) | Real time sharing method and device of images | |
| CN105022789A (en) | Plant species authentication method and system based on geographic position and feature image | |
| CN108737372A (en) | A kind of stream media playback method, computer equipment and readable medium | |
| CN110084698A (en) | Interactive system, exchange method and device based on block chain | |
| CN104899278B (en) | A kind of generation method and device of Hbase database data operation log | |
| CN105828017B (en) | A kind of cloud storage access system and method towards video conference | |
| CN105900176A (en) | System and method for resolving DRAM page conflicts based on memory access patterns | |
| CN106330668A (en) | WeChat-based voting implementation method and system | |
| CN106899577A (en) | A kind of Video service cloud platform for long-distance education | |
| CN103888711A (en) | Intelligent audio and video conference system | |
| CN107277146B (en) | Distributed storage service flow model generation method and system | |
| CN108765084A (en) | A kind of synchronization processing method and device of virtual three-dimensional space | |
| CN110377439A (en) | A kind of system of real time data acquisition push | |
| CN109951734A (en) | Present synthesis methods of exhibiting, system and storage medium based on dynamic video poster | |
| CN111951385A (en) | Enterprise display method and device | |
| Wang | Optimization of the online teaching system based on streaming media |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20191025 |
|
| RJ01 | Rejection of invention patent application after publication |