CN105956082A - Real-time data processing and storage system - Google Patents
Real-time data processing and storage system Download PDFInfo
- Publication number
- CN105956082A CN105956082A CN201610281670.4A CN201610281670A CN105956082A CN 105956082 A CN105956082 A CN 105956082A CN 201610281670 A CN201610281670 A CN 201610281670A CN 105956082 A CN105956082 A CN 105956082A
- Authority
- CN
- China
- Prior art keywords
- data
- module
- real time
- real
- time data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/54—Indexing scheme relating to G06F9/54
- G06F2209/541—Client-server
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Computing Systems (AREA)
- Data Mining & Analysis (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a real-time data processing system. Data filtering, processing and condition storage of data in a transmission process are realized in a Pub/Sub message queue topic mode based on a message queue data transmission channel constructed on a TCP or UDP transmission protocol. The system involved in the invention provides a flexible, dynamically deployable and pluggable real-time data processing framework, so that corresponding data processing systems can be quickly deployed and realized for different application environments and demands, services can be easily cloudized, scalability and expandability are achieved, the utilization efficiency of computing resources is improved, and the service development and deployment time is shortened.
Description
Technical field
The present invention relates to Internet of Things, big data technique field.Particularly with regard to Internet of Things data acquisition, locate in real time
Reason and memory technology.
Background technology
Big data are commonly defined as a kind of scale and arrive greatly obtaining, store, manage, are well beyond in terms of analysis
The data acquisition system of traditional database software means capability scope, has the data scale of magnanimity, quick data stream
Turn, various data type and the four big features such as value density is low.
The strategic importance of big data technique does not lies in grasps huge data message, and is these containing meaningful
Data carry out specialized process.In other words, if big data are compared to a kind of industry, then this industry
Realize the key of profit, be to improve " working ability " to data, realize " increasing of data by " processing "
Value " and build business model.
Technically, big data and the relation of cloud computing are inseparable just as the positive and negative of one piece of coin.
Big data necessarily cannot process with the computer of separate unit, it is necessary to uses distributed structure/architecture.Its characteristic is
Mass data is carried out distributed data digging.But it must rely on the distributed treatment of cloud computing, distributed number
According to storehouse and cloud storage, Intel Virtualization Technology.And calculating real time data processing in other words in real time is the one of big market demand
Individual basic link and key technology, be to realize quickly response, the key of high-speed decision of based on data.
Calculating in real time is typically all carried out for mass data, and general requirement is second level.Calculate main point in real time
It it is two pieces: the warehouse-in in real time of data, the real-time process of data.Calculating in real time of internet arena is typically all pin
Mass data is carried out, in addition to as the demand (as result of calculation is accurate) of non real-time calculating, counts in real time
Calculating a most important demand is real-time response result of calculation, typically requirement to be second level.
The application scenarios of real time data processing generally comprises:
1) data source is real-time continual, it is desirable to the response time of user be also real-time (such as
The stream data of large-scale website: what content the access PV/UV of website, user have accessed, searched in what
Holding, real-time data calculate and analyze and can refresh user accesses data with dynamic realtime, show that website is real-time
The situation of change of flow, analyzes flow and the user distribution situation of each hour of every day)
2) data volume is big and cannot or need not be it is expected that but require that the response time to user is real-time.Ratio
Such as the substantial amounts of rain sensor of somewhere distribution, owing to rainfall reaches certain threshold value, it is simultaneously emitted by alarm.
3) serve log stream, the various operands that industry or network system produce in continual running
According to and operation/maintenance data and it needs to based on these daily record datas, system carried out continual real-time monitoring and analysis.
Summary of the invention
It is an object of the invention to make full use of data system based on Pub/Sub message queue message transmission mode
Architecture design, optimizes the transmission of data, process and memory module, it is achieved real-time, efficient, flexible, expansible
Data process, storage and application system, provide Development Framework and base for Internet of Things, big market demand and service
Plinth ability, and real time data processing and storage system are provided.
The technical scheme that the present invention uses to achieve these goals is: real time data processing and storage system,
Described system is message transmission mode based on Pub/Sub, by data acquisition end, application end, process end, deposits
Storage end and displaying end are attached by the way of loose coupling, interactively with each other, form service-oriented system tray
Structure, described system includes Message Queuing Services module, application program module, Database Systems module, in real time
Data display module and data filter engine module,
Described Message Queuing Services module is a real-time data channel agency, and its realization can include based on communication protocol
AMQP, MQTT, Websocket and TCP, described Message Queuing Services module provides for real-time Data Transmission
Pipeline；
By system A interface, (wherein the implementation of A interface depends on message queue clothes to described application program module
The business communication protocol that used of device) realize input or the output of data, the main body bag that application program module is run
Include personal computer (PC), server, mobile terminal (such as smart mobile phone) and embedded device；
The described page program that real time data display module is a sing on web, it passes through HTTP, Websocket
Or MQ agreement obtains data, and real-time exhibition from Message Queuing Services device；
Described Database Systems module includes system R and NoSQL data base；
Described data filtering engine modules is that a shell script performs environment, described data filtering engine modules, institute
State data filtering engine modules program run time computer language include Javascript, Java,
Perl, Python, Scala and Erlang.
Further, described system also includes that the data of plug-in process card module, and described card module is straight
Connecing and act on the data of flowing in transmission channel, described card module is submitted to number by the way of Dynamical Deployment
According to filter engine module and run, card module is the program file after program source code or compiling.
Further, described data processing engine module subscribes to corresponding message subject according to demand, described in disappear
The data structure of breath theme includes a type (Type), client identification (ClientID) and semantic topic
(Topic)。
Further, in the data structure of described message subject, the connected mode of each several part includes:
1, connected by the character string of special interval character, such as " gateway_dsd2348_relay ", described
Special interval character include underscore " _ ", asterisk " * " and pound sign " # "；
2, by length and the composition of binary number segment identification each several part content.
Further, described data filtering engine modules is provided with data processor, described data processor
The executive mode of process include local function call, RPC (remote procedure call), REST API or
Webservice。
Further, described data processor according to the TAG comprised in each data carry out condition coupling and
Secondary filter, in described data processor, pending packet contains one group of TAG value, and the overall of data is tied
Structure includes that JSON form or XML format, described TAG value include numeral or character string；At described data
When reason program carries out filtering coupling to TAG value, including single TAG being mated and carrying out OR or AND
Mate Deng combination condition.
Further, the data after described data processor processes need when there is situations below to turn
Send out:
1, the TAG that packet contains there is a need to the mark being forwarded, such as " relay_to_ "；
2, determined by the logic of data processor, as certain type of data need to forward, or after process
Data meet corresponding condition (such as larger than certain threshold value).
Further, the data after described data processor processes need when there is situations below to write number
According to storehouse:
1, the TAG that packet contains there is a need to be stored in the mark of data base, such as " DB "；
2, determined by the logic of data processor, as the data in resulting from certain period when data need write
Data base, and the data of other times section need not write.
Technique scheme is used to provide the benefit that: present invention simultaneous asynchronous data transmissions based on message queue is led to
Road, by the message subject pattern of Pub/Sub, it is achieved data data filtering in transmitting procedure, process and
Condition stores；Propose one and carry out real time data read-write, and integrated conventional relationship based on NoSQL data-base cluster
The system framework of type data base, it is achieved to left over by history data or the compatibility of system, and realize based on SQL
The compatibility of the application and development of inquiry.The invention provides a kind of flexibly, can Dynamical Deployment, pluggable reality
Time data processing shelf, it is simple to for different applied environments and demand, rapid deployment and realize corresponding data
Processing system so that be prone to service cloud, becomes scalable, expansible, improves the use effect calculating resource
Rate and shortening service development and deployment time.
Accompanying drawing explanation
Fig. 1 is real time data processing involved in the present invention and storage system block diagram；
Fig. 2 is the data filtering of real time data processing involved by the present invention and storage system and processes application framework；
Fig. 3 is data filtering involved by the present invention and processing procedure schematic diagram；
Fig. 4 is the data structure schematic diagram of system message theme；
Fig. 5 is the message subject data structure schematic diagram of Pub/Sub involved in the present invention；
Fig. 6 is the data filtering engine involved in the present invention schematic diagram one to data filtering；
Fig. 7 is the data filtering engine involved in the present invention schematic diagram two to data filtering；
Fig. 8 is the data filtering engine involved in the present invention schematic diagram three to data filtering；
Fig. 9 is the schematic diagram of transmission data involved in the present invention.
Detailed description of the invention
Below in conjunction with accompanying drawing and currently preferred specific embodiment, present disclosure is said further
Bright.Example is served only for explaining the present invention, is not intended to limit the scope of the present invention.
As shown in Fig. 1 to Fig. 9, the real time data processing described in this programme and storage system, the present invention
Involved system is message transmission mode based on Pub/Sub, by data acquisition end, application end, process end,
Storage end and displaying end are attached by the way of loose coupling, interactively with each other, form service-oriented system
Framework so that whole system be one flexibly, be prone to the service development framework of extension, and realize one in real time
Data handling system, including at: real time data processing and storage system and application framework and real time data thereof
Reason and the mode filtered；System described in this programme is as it is shown in figure 1, system involved in the present invention is constituted
Including: Message Queuing Services apparatus module, application program module, Database Systems module, real time data are shown
Module and data filter engine module.Wherein, Message Queuing Services apparatus module is a real-time data channel generation
Reason, its realization is to include AMQP, MQTT, Websocket and TCP etc., described message based on communication protocol
Queue service apparatus module is the core component of system involved in the present invention, provides pipeline for real-time Data Transmission；
By system A interface, (wherein the implementation of A interface depends on message queue clothes to described application program module
The business communication protocol that used of device) realize input or the output of data, the main body bag that application program module is run
Include personal computer (PC), server, mobile terminal (such as smart mobile phone) and embedded device etc., typically transport
Row application program on embedded device (such as host computer), as the collection terminal of data, inputs data toward system；
And operate in the use end as data of the application program on PC or server.Real time data display module is one
The page program of individual sing on web, it is filled from Message Queuing Services by HTTP, Websocket or MQ agreement
Put acquisition data, and real-time exhibition；Data filtering engine modules is that a shell script performs environment, data mistake
Filter engine modules is that a main modular of system involved by this programme, major part real time data processing and filtration will
Carrying out in this data filtering engine modules, the program runtime environment of this data filtering engine modules includes
The computer languages such as Javascript, Java, Perl, Python, Scala and Erlang, various operations
Time environment itself and operation principle implement, according to system, the program runtime environment that uses or support, this
The involved real time data processing of invention will be realized by corresponding development language with the logic of filtration；Database Systems mould
Block inclusion relation type Database Systems and NoSQL data base, but in this programme, by interface D and message team
The mutual data base of row service unit is only limitted to NoSQL data base, in order to ensure real-time that data transmit and deposit
The extensibility of energy storage power, system involved by this programme uses NoSQL data-base cluster directly to take with message queue
Business device is mutual, and which kind of NoSQL Database Systems this programme specifically chosen is not restricted.System is implementing
Time according to selected NoSQL Database Systems, in the NoSQL data of the integrated correspondence of message queue apparatus module
Storehouse client, as shown in Figure 1.
Real time data processing involved in this programme and storage system also provide for a kind of pluggable data and process
Card module, as in figure 2 it is shown, each card module application program directly acts in transmission logical in this programme
The data of flowing in road, each plug-in applications is submitted to data filtering engine mould by the way of Dynamical Deployment
Block also runs (Fig. 3), and card module application program can be the program file after program source code or compiling.
In system described in this programme, collected or input data pass through Message Queuing Services device mould
Block, according to condition write direct NoSQL data-base cluster or processed by other filter inserts, or directly quilt
Neglect.If as it is shown on figure 3, system needs to deposit in dependence system carrying out data filtering or process when
Data carry out logical judgment, data filtering plug-in unit is directly by the number in data read and write interface reading database
According to.In this programme, involved system support service logic based on relevant database (RDBMS) judges and base
Data query and application in SQL.If system history of existence Legacy Data is at traditional Relational DataBase
In, system needs to inquire about the data in Already in relevant database or relation when carrying out data filtering,
Timing (by the second, point or other times unit, concrete system depending on realizing) or trigger according to event
By the data syn-chronization in relevant database to NoSQL data-base cluster；It is stored in NoSQL data-base cluster
Data processed result be synchronized in relevant database by Data synchronization interface also according to needs.
Mode that data described in this programme process as it is shown on figure 3, the most each application filtering according to
Demand, subscribes to certain message subject, and the data structure definition of each message subject is as shown in Figure 4, comprises:
One type (Type), client identification (ClientID) and semantic topic (Topic) etc., each message
In subject data structure, the connected mode of each several part includes but following several ways:
1, connected by the character string of special interval character, such as " gateway_dsd2348_relay ", and even
Connect spcial character include but not limited to underscore " _ ", asterisk " * ", pound sign " # " etc.；
2, by length and the composition of binary number segment identification each several part content, as shown in Figure 5.
In this programme when client to send the data about certain semantic topic (Topic), client
Application program can assemble them into the data structure of system message theme automatically, and other parts in data structure
Can read from equipment or client terminal attribute, it is also possible to actively specified by programming.Then, application program
By system interface A (implementing selected communication protocol according to system), data are issued (Pub) arrive
The above-mentioned message subject assembled, is received by Message Queuing system's device, is transmitted to data filtering engine, and
Being filtered according to message subject by data filtering engine, as shown in Figure 6, filter engine is respectively for the mode of filtration
The various piece of message subject is mated (Fig. 6), or each several part is carried out OR (logic or) condition
Combinations matches (Fig. 7), or each several part is carried out AND (logical AND) conditional combination coupling (Fig. 8), with
And the combinations matches of various mixing.Match pattern also includes arbitrarily certain portion of (coupling entirely) pattern, i.e. message subject
Divide or some is that arbitrary value is all by successful match.According to different matching results, data filtering engine performs
Corresponding data processor (such as Fig. 6, Fig. 7 and Fig. 8), the execution side of data processor process
Formula includes but local function call, RPC (remote procedure call), REST API and Webservice etc..
Data processor carries out condition coupling and secondary filter according to the TAG comprised in each data, such as Fig. 9
Shown in, in system involved in the present invention, every pending data all comprise one group of TAG value, and the entirety of data
Structure can be a JSON form and XML format, naturally it is also possible to is other data representation forms；And TAG
Value can be numeral, character string or other data structure bodies；TAG value is carried out filtering coupling by data processor
Time, both single TAG can be mated, it is also possible to carry out the combination condition couplings such as OR or AND.
In system involved in the present invention, data base or forwarding can be written into for the data after routine processes
/ it is routed to other processing routines or client application, if desired forward, these data will be published (Pub)
To the message subject (as shown by dotted lines in figure 3) of one or more correspondences, and by have subscribed the client of corresponding theme
End application program or other processing routines receive.Data after process have following two to determine the need of forwarding
Mode:
1), the TAG that contains of packet there is a need to the mark that is forwarded, such as " relay_to_ "；
2), determined by the logic of data processor, as certain type of data need to forward, or after process
Data meet certain condition (such as larger than certain threshold value) etc..
System involved in this programme, for data the need of write into Databasce, is processed journey by data
Sequence determines, and write according to comprising following two situation:
1) TAG that packet contains there is a need to be stored in the mark of data base, such as " DB "；
2) determined by the logic of data processor, as the data in resulting from certain period when data need write
Data base, and the data of other times section need not write etc..
The foregoing is only presently preferred embodiments of the present invention, not in order to limit the present invention, all essences in the present invention
Within god and principle, any modification, equivalent substitution and improvement etc. made, should be included in the protection of the present invention
Within the scope of.
Claims (8)
1. real time data processing and storage system, described system is message transmission mode based on Pub/Sub, by number
It is attached by the way of loose coupling according to collection terminal, application end, process end, storage end and displaying end, that
This is mutual, forms service-oriented system architecture, it is characterised in that: described system includes Message Queuing Services
Module, application program module, Database Systems module, real time data display module and data filter engine module,
Described Message Queuing Services module is a real-time data channel agency, and its realization can include based on communication protocol
AMQP, MQTT, Websocket and TCP, described Message Queuing Services module provides for real-time Data Transmission
Pipeline；
By system A interface, (wherein the implementation of A interface depends on message queue clothes to described application program module
The business communication protocol that used of device) realize input or the output of data, the main body bag that application program module is run
Include personal computer (PC), server, mobile terminal (such as smart mobile phone) and embedded device；
The described page program that real time data display module is a sing on web, it passes through HTTP, Websocket
Or MQ agreement obtains data, and real-time exhibition from Message Queuing Services device；
Described Database Systems module includes system R and NoSQL data base；
Described data filtering engine modules is that a shell script performs environment, described data filtering engine modules, institute
State data filtering engine modules program run time computer language include Javascript, Java,
Perl, Python, Scala and Erlang.
2. according to the real time data processing described in claim 1 and storage system, it is characterised in that: described system
Also including that the data of plug-in process card module, described card module directly acts on and flows in transmission channel
Dynamic data, described card module is submitted to data filtering engine modules by the way of Dynamical Deployment and runs,
Card module is the program file after program source code or compiling.
3. according to the real time data processing described in claim 1 and storage system, it is characterised in that: described number
Subscribing to corresponding message subject according to demand according to processing engine modules, the data structure of described message subject includes one
Individual type (Type), client identification (ClientID) and semantic topic (Topic).
4. according to the real time data processing described in claim 3 and storage system, it is characterised in that: described message
In the data structure of theme, the connected mode of each several part includes:
4-1, connected by the character string of special interval character, such as " gateway_dsd2348_relay ", described
Special interval character include underscore " _ ", asterisk " * " and pound sign " # "；
4-2, by the length of binary number segment identification each several part content and composition.
5. according to the real time data processing described in claim 1 and storage system, it is characterised in that: described data
Filter engine module is provided with data processor, and the executive mode of described data processor process includes
Local function call, RPC (remote procedure call), REST API or Webservice.
6. according to the real time data processing described in claim 5 and storage system, it is characterised in that: described number
Condition coupling and secondary filter is carried out according to the TAG comprised in each data, at described data according to processing routine
In reason program, pending packet contains one group of TAG value, and the overall structure of data includes JSON form or XML
Form, described TAG value includes numeral or character string；TAG value is filtered by described data processor
During coupling, including single TAG being mated and carrying out the combination condition couplings such as OR or AND.
7. according to the real time data processing described in claim 6 and storage system, it is characterised in that: described data
Data after routine processes need when there is situations below to forward:
The TAG that 7-1, packet contain there is a need to the mark being forwarded, such as " relay_to_ "；
7-2, by data processor logic determine, as certain type of data need forward, or process after number
According to meeting corresponding condition (such as larger than certain threshold value).
8. according to the real time data processing described in claim 6 and storage system, it is characterised in that: described data
Data after routine processes need write into Databasce when there is situations below:
The TAG that 8-1, packet contain there is a need to be stored in the mark of data base, such as " DB "；
8-2, being determined by the logic of data processor, the data in such as resulting from certain period when data need write
Data base, and the data of other times section need not write.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610281670.4A CN105956082B (en) | 2016-04-29 | 2016-04-29 | Real time data processing and storage system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610281670.4A CN105956082B (en) | 2016-04-29 | 2016-04-29 | Real time data processing and storage system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN105956082A true CN105956082A (en) | 2016-09-21 |
| CN105956082B CN105956082B (en) | 2019-07-02 |
Family
ID=56913056
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610281670.4A Active CN105956082B (en) | 2016-04-29 | 2016-04-29 | Real time data processing and storage system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN105956082B (en) |
Cited By (19)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107241422A (en) * | 2017-06-23 | 2017-10-10 | 郑州云海信息技术有限公司 | It is a kind of in real time by the method for external user and subscriber group information with stepping Apache Ranger |
| CN107480283A (en) * | 2017-08-23 | 2017-12-15 | 九次方大数据信息集团有限公司 | Realize the method, apparatus and storage system of big data quick storage |
| CN107766538A (en) * | 2017-10-28 | 2018-03-06 | 杭州安恒信息技术有限公司 | Data filtering processing module and synchronous, asynchronous filter method based on java |
| CN107959699A (en) * | 2016-10-18 | 2018-04-24 | 中车株洲电力机车研究所有限公司 | A kind of WebSocket data delivery systems and method based on Stream Processing |
| CN108133062A (en) * | 2018-02-02 | 2018-06-08 | 北京火花日志科技有限公司 | A kind of data processing system |
| CN108255802A (en) * | 2016-12-29 | 2018-07-06 | 北京国双科技有限公司 | Generic text Analytical framework and the method and apparatus based on framework parsing text |
| CN108268329A (en) * | 2017-12-28 | 2018-07-10 | 武汉长江仪器自动化研究所有限公司 | A kind of data processing method and device of the grouting automatic recorder network terminal |
| CN108304267A (en) * | 2018-01-31 | 2018-07-20 | 中科边缘智慧信息科技（苏州）有限公司 | The multi-source data of highly reliable low-resource expense draws the method for connecing |
| CN108614820A (en) * | 2016-12-09 | 2018-10-02 | 腾讯科技（深圳）有限公司 | The method and apparatus for realizing the parsing of streaming source data |
| CN109684416A (en) * | 2018-11-13 | 2019-04-26 | 国电南京自动化股份有限公司 | A kind of high concurrent real-time history data storage system |
| CN109831385A (en) * | 2019-01-22 | 2019-05-31 | 北京奇艺世纪科技有限公司 | A kind of message treatment method, device and electronic equipment |
| CN110109996A (en) * | 2019-05-15 | 2019-08-09 | 浪潮通用软件有限公司 | A kind of industry internet equipment real-time data acquisition and storage method and system |
| CN110537360A (en) * | 2017-04-19 | 2019-12-03 | 微软技术许可有限责任公司 | Internet of Things PUB-SUB data publisher |
| CN111309335A (en) * | 2020-02-28 | 2020-06-19 | 腾讯音乐娱乐科技（深圳）有限公司 | Plug-in application compiling method and device and computer readable storage medium |
| CN111428128A (en) * | 2020-02-20 | 2020-07-17 | 北京达佳互联信息技术有限公司 | Method for generating pluggable recommendation system, service recommendation method, device and equipment |
| CN112073506A (en) * | 2020-09-04 | 2020-12-11 | 哈尔滨工业大学 | IPv6 and message queue-based complex electromagnetic data acquisition method |
| CN112422613A (en) * | 2020-09-16 | 2021-02-26 | 北京中兵数字科技集团有限公司 | Data processing method, data processing platform and computer readable storage medium |
| CN112995263A (en) * | 2019-12-18 | 2021-06-18 | 中国移动通信集团陕西有限公司 | Network priority data processing system |
| CN113220720A (en) * | 2021-06-08 | 2021-08-06 | 上海天旦网络科技发展有限公司 | Real-time second-level data processing and query method and system |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1959717A (en) * | 2006-10-09 | 2007-05-09 | 北京道达天际软件技术有限公司 | System and method for preprocessing mass remote sensing data collection driven by order form |
| CN103164721A (en) * | 2013-03-15 | 2013-06-19 | 安徽皖通科技股份有限公司 | Car connected network side information data collection and processing method |
| CN104144071A (en) * | 2013-05-10 | 2014-11-12 | 北京新媒传信科技有限公司 | System log processing method and platform |
| CN104572975A (en) * | 2014-12-31 | 2015-04-29 | 中国石油天然气股份有限公司 | Real-time data processing analysis system |
| CN105512939A (en) * | 2015-12-04 | 2016-04-20 | 中国建设银行股份有限公司 | Foreign exchange transaction-related data storage and query method, declaration method and system |
-
2016
- 2016-04-29 CN CN201610281670.4A patent/CN105956082B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1959717A (en) * | 2006-10-09 | 2007-05-09 | 北京道达天际软件技术有限公司 | System and method for preprocessing mass remote sensing data collection driven by order form |
| CN103164721A (en) * | 2013-03-15 | 2013-06-19 | 安徽皖通科技股份有限公司 | Car connected network side information data collection and processing method |
| CN104144071A (en) * | 2013-05-10 | 2014-11-12 | 北京新媒传信科技有限公司 | System log processing method and platform |
| CN104572975A (en) * | 2014-12-31 | 2015-04-29 | 中国石油天然气股份有限公司 | Real-time data processing analysis system |
| CN105512939A (en) * | 2015-12-04 | 2016-04-20 | 中国建设银行股份有限公司 | Foreign exchange transaction-related data storage and query method, declaration method and system |
Cited By (29)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107959699A (en) * | 2016-10-18 | 2018-04-24 | 中车株洲电力机车研究所有限公司 | A kind of WebSocket data delivery systems and method based on Stream Processing |
| CN108614820B (en) * | 2016-12-09 | 2021-01-15 | 腾讯科技（深圳）有限公司 | Method and device for realizing streaming source data analysis |
| CN108614820A (en) * | 2016-12-09 | 2018-10-02 | 腾讯科技（深圳）有限公司 | The method and apparatus for realizing the parsing of streaming source data |
| CN108255802A (en) * | 2016-12-29 | 2018-07-06 | 北京国双科技有限公司 | Generic text Analytical framework and the method and apparatus based on framework parsing text |
| CN108255802B (en) * | 2016-12-29 | 2021-08-24 | 北京国双科技有限公司 | Universal text parsing architecture and method and device for parsing text based on architecture |
| CN110537360B (en) * | 2017-04-19 | 2022-06-14 | 微软技术许可有限责任公司 | Internet of things PUB-SUB data publisher |
| CN110537360A (en) * | 2017-04-19 | 2019-12-03 | 微软技术许可有限责任公司 | Internet of Things PUB-SUB data publisher |
| CN107241422B (en) * | 2017-06-23 | 2020-08-11 | 浪潮云信息技术股份公司 | Method for synchronizing external user and user group information into Apache Range in real time |
| CN107241422A (en) * | 2017-06-23 | 2017-10-10 | 郑州云海信息技术有限公司 | It is a kind of in real time by the method for external user and subscriber group information with stepping Apache Ranger |
| CN107480283A (en) * | 2017-08-23 | 2017-12-15 | 九次方大数据信息集团有限公司 | Realize the method, apparatus and storage system of big data quick storage |
| CN107766538A (en) * | 2017-10-28 | 2018-03-06 | 杭州安恒信息技术有限公司 | Data filtering processing module and synchronous, asynchronous filter method based on java |
| CN108268329A (en) * | 2017-12-28 | 2018-07-10 | 武汉长江仪器自动化研究所有限公司 | A kind of data processing method and device of the grouting automatic recorder network terminal |
| CN108268329B (en) * | 2017-12-28 | 2022-01-28 | 武汉长江仪器自动化研究所有限公司 | Data processing method and device for grouting recorder network terminal |
| CN108304267A (en) * | 2018-01-31 | 2018-07-20 | 中科边缘智慧信息科技（苏州）有限公司 | The multi-source data of highly reliable low-resource expense draws the method for connecing |
| CN108304267B (en) * | 2018-01-31 | 2022-03-22 | 中科边缘智慧信息科技（苏州）有限公司 | Multi-source data leading method with high reliability and low resource overhead |
| CN108133062A (en) * | 2018-02-02 | 2018-06-08 | 北京火花日志科技有限公司 | A kind of data processing system |
| CN109684416A (en) * | 2018-11-13 | 2019-04-26 | 国电南京自动化股份有限公司 | A kind of high concurrent real-time history data storage system |
| CN109831385A (en) * | 2019-01-22 | 2019-05-31 | 北京奇艺世纪科技有限公司 | A kind of message treatment method, device and electronic equipment |
| CN109831385B (en) * | 2019-01-22 | 2021-11-05 | 北京奇艺世纪科技有限公司 | Message processing method and device and electronic equipment |
| CN110109996A (en) * | 2019-05-15 | 2019-08-09 | 浪潮通用软件有限公司 | A kind of industry internet equipment real-time data acquisition and storage method and system |
| CN112995263A (en) * | 2019-12-18 | 2021-06-18 | 中国移动通信集团陕西有限公司 | Network priority data processing system |
| CN111428128B (en) * | 2020-02-20 | 2021-09-10 | 北京达佳互联信息技术有限公司 | Method for generating pluggable recommendation system, service recommendation method, device and equipment |
| CN111428128A (en) * | 2020-02-20 | 2020-07-17 | 北京达佳互联信息技术有限公司 | Method for generating pluggable recommendation system, service recommendation method, device and equipment |
| CN111309335A (en) * | 2020-02-28 | 2020-06-19 | 腾讯音乐娱乐科技（深圳）有限公司 | Plug-in application compiling method and device and computer readable storage medium |
| CN111309335B (en) * | 2020-02-28 | 2023-08-15 | 腾讯音乐娱乐科技（深圳）有限公司 | Compiling method and device of plug-in application and computer readable storage medium |
| CN112073506A (en) * | 2020-09-04 | 2020-12-11 | 哈尔滨工业大学 | IPv6 and message queue-based complex electromagnetic data acquisition method |
| CN112422613A (en) * | 2020-09-16 | 2021-02-26 | 北京中兵数字科技集团有限公司 | Data processing method, data processing platform and computer readable storage medium |
| CN112422613B (en) * | 2020-09-16 | 2022-02-01 | 北京中兵数字科技集团有限公司 | Data processing method, data processing platform and computer readable storage medium |
| CN113220720A (en) * | 2021-06-08 | 2021-08-06 | 上海天旦网络科技发展有限公司 | Real-time second-level data processing and query method and system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN105956082B (en) | 2019-07-02 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN105956082A (en) | Real-time data processing and storage system | |
| CN104182389B (en) | A kind of big data analyzing business intelligence service system based on semanteme | |
| CN109857917A (en) | Towards the security knowledge map construction method and system for threatening information | |
| CN110784419A (en) | Method and system for visualizing professional data of railway electric affairs | |
| CN108365971A (en) | Daily record analytic method, equipment and computer-readable medium | |
| CN106547766A (en) | A kind of data access method and device | |
| D'silva et al. | Real-time processing of IoT events with historic data using Apache Kafka and Apache Spark with dashing framework | |
| CN109492012A (en) | A kind of method, apparatus and storage medium of data real-time statistics and retrieval | |
| CN108572963A (en) | Information acquisition method and device | |
| CN107241305B (en) | Network protocol analysis system based on multi-core processor and analysis method thereof | |
| CN110795697B (en) | Method and device for acquiring logic expression, storage medium and electronic device | |
| CN109547474A (en) | A kind of data transmission method and device | |
| CN104301436A (en) | Push method and device of content to be displayed, subscription method and device of content to be displayed and update method and device of content to be displayed | |
| CN107689999A (en) | A kind of full-automatic computational methods of cloud platform and device | |
| CN108038207A (en) | A kind of daily record data processing system, method and server | |
| CN111538774B (en) | Data storage and display method, system, equipment and storage medium | |
| CN106250159B (en) | Mixed programming method using natural language | |
| CN110019087A (en) | Data processing method and its system | |
| CN104268143A (en) | XML (extensible markup language) data processing method and device | |
| CN108073625A (en) | For the system and method for metadata information management | |
| WO2012114284A1 (en) | Peer-to-peer collaboration of publishers in a publish-subscription environment | |
| Kshetri et al. | Big data and cloud computing for development: Lessons from key industries and economies in the global south | |
| CN104199889A (en) | RTLogic big data processing system and method based on CEP technology | |
| CN104021216B (en) | Message proxy server and information publish subscription method and system | |
| CN104834730B (en) | data analysis system and method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB02 | Change of applicant information |
Address after: 518000 B502, light Science Park, No. 7 West West Road, Shenzhen, Guangdong, Nanshan District Applicant after: Shenzhen great number of Technology Co., Ltd. Address before: 518000 B502, light Science Park, No. 7 West West Road, Shenzhen, Guangdong, Nanshan District Applicant before: SHENZHEN QIANHAI DASHUDIAN TECHNOLOGY CO., LTD. |
|
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20210108 Address after: 518000 Guangdong city of Shenzhen province Nanshan District South Road No. 10 four Patentee after: Liu Fujianbo Address before: 518000 B502, Languang Science Park, No.7, Xinxi Road, Nanshan District, Shenzhen City, Guangdong Province Patentee before: Shenzhen great number of Technology Co.,Ltd. |
|
| TR01 | Transfer of patent right | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20210331 Address after: 518000 f3.8-8b4 (b), Tianjing building, Tian'an chegongmiao Industrial Zone, Shatou street, Futian District, Shenzhen City, Guangdong Province Patentee after: SHENZHEN ZHIYUAN YOUXUE EDUCATION TECHNOLOGY Co.,Ltd. Address before: 518000 Guangdong city of Shenzhen province Nanshan District South Road No. 10 four Patentee before: Liu Fujianbo |