CN105956082B - Real time data processing and storage system - Google Patents
Real time data processing and storage system Download PDFInfo
- Publication number
- CN105956082B CN105956082B CN201610281670.4A CN201610281670A CN105956082B CN 105956082 B CN105956082 B CN 105956082B CN 201610281670 A CN201610281670 A CN 201610281670A CN 105956082 B CN105956082 B CN 105956082B
- Authority
- CN
- China
- Prior art keywords
- data
- real time
- module
- message
- data processing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/54—Indexing scheme relating to G06F9/54
- G06F2209/541—Client-server
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Computing Systems (AREA)
- Data Mining & Analysis (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The present invention discloses a kind of real-time data processing system, message queue data transmission channel based on building on TCP or UDP transport protocol, by the message queue subject pattern of Pub/Sub, data filtering, processing and condition storage of the data in transmission process are realized.System involved in the present invention provide it is a kind of flexible, can Dynamical Deployment, pluggable real time data processing frame, convenient for being directed to different application environment and demand, rapid deployment and the corresponding data processing system of realization, it makes it easy to service cloud, become scalable, expansible, improve the service efficiency of computing resource and shortens service development and deployment time.
Description
Technical field
The present invention relates to Internet of Things, big data technical field.Particularly with regard to Internet of Things data acquisition, in real time handle and deposit
Storage technology.
Background technique
Big data is commonly defined as a kind of scale and arrives well beyond traditional number in terms of acquisition, storage, management, analysis greatly
According to the data acquisition system of library software means capability range, the data class of data scale, quick stream compression, multiplicity with magnanimity
The four big features such as type and value density are low.
The strategic importance of big data technology, which is not lain in, grasps huge data information, and is to these containing significant number
According to progress specialized process.In other words, if big data is compared to a kind of industry, this industry realizes the pass of profit
Key is to improve " working ability " to data, realizes " increment " of data by " processing " and construct business model.
Technically, big data is inseparable just as the front and back sides of one piece of coin with the relationship of cloud computing.Big number
According to can not necessarily be handled with the computer of separate unit, it is necessary to use distributed structure/architecture.Its characteristic be to mass data into
Row distributed data digging.But it must rely on distributed treatment, distributed data base and the cloud storage of cloud computing, virtualization skill
Art.And a basic link and key technology that real time data processing in other words is big data application are calculated in real time, it is to realize base
Quick response, the key of high-speed decision in data.
Calculating in real time is typically all to carry out for mass data, general to require to be second grade.It calculates in real time and is broadly divided into two
Block: the real-time processing of the real-time storage, data of data.The real-time calculating of internet area is typically all to carry out for mass data
, other than the demand (such as calculated result is accurate) as non real-time calculating, calculate in real time a most important demand be can
Real-time response calculated result is general to require to be second grade.
The application scenarios of real time data processing generally comprise:
1) data source is continual in real time, it is desirable that the response time of user is also in real time (such as large-scale net
The stream data stood: access PV/UV, the user of website have accessed any content, have searched for what content etc., real-time data meter
Calculation and analysis can with dynamic realtime refresh user accesses data, show the situation of change of website real-time traffic, and analysis is each daily
The flow and user distribution situation of hour)
2) data volume is big and can not or need not be, it is expected that but requiring to be real-time to the response time of user.Such as somewhere
The a large amount of precipitation rain fall sensor of area's distribution is simultaneously emitted by alarm since rainfall reaches certain threshold value.
3) the various operation datas and fortune that serve log stream, industry or network system generate in continual operational process
Dimension data, and need to carry out continual real time monitoring and analysis to system based on these daily record datas.
Summary of the invention
The purpose of the present invention is make full use of the data system framework based on Pub/Sub message queue message transmission mode to set
Meter optimizes the transmission, processing and memory module of data, realizes real-time, efficient, flexible, expansible data processing, stores and answer
With system, Development Framework and base power are provided for Internet of Things, big data application and service, and real time data processing is provided and is deposited
Storage system.
The technical solution that the present invention uses to achieve the goals above are as follows: real time data processing and storage system, it is described
System is the message transmission mode based on Pub/Sub, by data collection terminal, application end, processing end, storage end and shows that end leads to
The mode for crossing loose coupling is attached, interactively with each other, forms service-oriented system architecture, and the system includes message queue
Service module, application program module, Database Systems module, real time data display module and data filter engine module,
The Message Queuing Services module is a real-time data channel agency, and realization can include based on communication protocol
AMQP, MQTT, Websocket and TCP, the Message Queuing Services module provide pipeline for real-time Data Transmission；
By system A interface, (wherein the implementation of A interface depends on Message Queuing Services to the application program module
Communication protocol used by device) realize data input or output, application program module operation main body include individual calculus
Machine (PC), server, mobile terminal (such as smart phone) and embedded device；
The real time data display module is a page program based on Web, by HTTP, Websocket or
MQ agreement obtains data, and real-time exhibition from Message Queuing Services device；
The Database Systems module includes system R and NoSQL database；
The data filtering engine modules are a shell script performing environment, the data filtering engine modules, institute
State data filtering engine modules program operation when computer language include Javascript, Java, Perl,
Python, Scala and Erlang.
Further, the system also includes the data processing card module of plug-in, the card module is directly made
Data for flowing in transmission channel, the card module are submitted to data filtering engine mould by way of Dynamical Deployment
Block is simultaneously run, and card module is the program file after program source code or compiling.
Further, the data processing engine module subscribes to corresponding message subject, the message master according to demand
The data structure of topic includes a type (Type), client identification (ClientID) and semantic topic (Topic).
Further, the connection type of each section includes: in the data structure of the message subject
1, it is connected by the character string of special interval character, such as " gateway_dsd2348_relay ", described special
Every character include underscore " _ ", asterisk " * " and pound sign " # "；
2, pass through the length and composition of binary number segment identification each section content.
Further, the data filtering engine modules are equipped with data processor, the data processor
The executive mode of process includes local function call, RPC (remote procedure call), REST API or Webservice.
Further, the data processor according to include in each data TAG carry out condition matching and it is secondary
It filters, pending data includes one group of TAG value in the data processor, and the overall structure of data includes JSON format
Or XML format, the TAG value include number or character string；The data processor is filtered matching to TAG value
When, including the matching of the combination conditions such as OR or AND is matched and carried out to single TAG.
Further, the data after the data processor processing need to forward when there is following situations:
1, the mark in need being forwarded in the TAG that data include, such as " relay_to_ "；
2, it is determined by the logic of data processor, such as certain type of data need to forward, or treated data are full
The corresponding condition (such as larger than some threshold value) of foot.
Further, the data after the data processor processing need that data are written when there is following situations
Library:
1, the mark in need for being stored in database in the TAG that data include, such as " DB "；
2, it is determined by the logic of data processor, as when data generate the data in Mr. Yu's section time needed that data are written
Library, and the data of other times section do not need to be written.
Beneficial effect by adopting the above technical scheme is: the present invention is based on the nonsynchronous data transmission channels of message queue, lead to
The message subject mode of Pub/Sub is crossed, realizes data filtering, processing and condition storage of the data in transmission process；It is proposed one
Kind carries out real time data read-write, and the system framework of integrated traditional Relational DataBase, realization pair based on NoSQL data-base cluster
The compatibility of left over by history data or system, and realize the compatibility to the application and development based on SQL query.The present invention provides one kind
Flexibly, can Dynamical Deployment, pluggable real time data processing frame, convenient for being directed to different application environment and demand, fastly
Speed disposes and realizes corresponding data processing system, makes it easy to service cloud becoming scalable, expansible, improves and calculates
The service efficiency and shortening service development of resource and deployment time.
Detailed description of the invention
Fig. 1 is real time data processing involved in the present invention and storage system block diagram；
Fig. 2 is the data filtering and processing application framework of real time data processing involved by the present invention and storage system；
Fig. 3 is data filtering involved by the present invention and treatment process schematic diagram；
Fig. 4 is the data structure schematic diagram of system message theme；
Fig. 5 is the message subject data structure schematic diagram of Pub/Sub involved in the present invention；
Fig. 6 is schematic diagram one of the data filtering engine involved in the present invention to data filtering；
Fig. 7 is schematic diagram two of the data filtering engine involved in the present invention to data filtering；
Fig. 8 is schematic diagram three of the data filtering engine involved in the present invention to data filtering；
Fig. 9 is the schematic diagram of transmission data involved in the present invention.
Specific embodiment
The contents of the present invention are further described below in conjunction with attached drawing and currently preferred specific embodiment.It is lifted
Example is served only for explaining the present invention, is not intended to limit the scope of the present invention.
As shown in Fig. 1 to Fig. 9, real time data processing described in this programme and storage system are according to the present invention
System is the message transmission mode based on Pub/Sub, by data collection terminal, application end, processing end, storage end and shows that end leads to
The mode for crossing loose coupling is attached, interactively with each other, forms service-oriented system architecture, so that whole system is one flexible
, the service development frame for being easy to extend, and realize a real-time data processing system, including: real time data processing with
The mode of storage system and its application framework and real time data processing and filtering；System described in this programme as shown in Figure 1,
System according to the present invention composition include: Message Queuing Services apparatus module, application program module, Database Systems module,
Real time data display module and data filter engine module.Wherein, Message Queuing Services apparatus module is that a real time data is logical
Road agency, it includes AMQP, MQTT, Websocket and TCP etc., the Message Queuing Services that realization, which is based on communication protocol,
Apparatus module is the core component of system involved in the present invention, provides pipeline for real-time Data Transmission；The application program mould
Block is real by system A interface (wherein the implementation of A interface is dependent on communication protocol used by Message Queuing Services device)
The main body of the input or output of existing data, application program module operation includes personal computer (PC), server, mobile terminal
(such as smart phone) and embedded device generally operate in the application program on embedded device (such as host computer) as data
Collection terminal, toward system input data；And operate in use end of the application program on PC or server as data.Number in real time
It is a page program based on Web according to display module, passes through HTTP, Websocket or MQ agreement from Message Queuing Services
Device obtains data, and real-time exhibition；Data filtering engine modules are a shell script performing environment, data filtering engine mould
Block is a main modular of system involved by this programme, and most of real time data processing and filtering will be in the data filtering engines
Module carry out, the program runtime environment of the data filtering engine modules include Javascript, Java, Perl, Python,
The computer languages such as Scala and Erlang, various runtime environments itself and its working principle are adopted according to system specific implementation
With or support program runtime environment, real time data processing involved in the present invention will be by corresponding exploitation language with the logic of filtering
Speech is realized；Database Systems module inclusion relation type Database Systems and NoSQL database, but in the present solution, passing through interface D
The database interacted with Message Queuing Services device is only limitted to NoSQL database, in order to guarantee the real-time of data transmission and deposit
The scalability of energy storage power, system involved by this programme are directly handed over Message Queuing Services device using NoSQL data-base cluster
Mutually, it is specifically chosen which kind of NoSQL Database Systems this programme with no restriction.System is in specific implementation according to selected
NoSQL Database Systems integrate corresponding NoSQL database client in message queue apparatus module, as shown in Figure 1.
Real time data processing involved in this programme and storage system also provide a kind of pluggable data processing plug-in unit
Module, as shown in Fig. 2, each card module application program directly acts on the number flowed in transmission channel in this programme
According to each plug-in applications are submitted to data filtering engine modules by way of Dynamical Deployment and run (Fig. 3), plug-in unit mould
Block application program can be the program file after program source code or compiling.
In system described in this programme, it is collected or the data of input passes through Message Queuing Services apparatus module, quilt
It according to condition writes direct NoSQL data-base cluster or is handled by other filter inserts, or be directly ignored.Such as Fig. 3 institute
Show, if system needs already present data in dependence system to carry out logic judgment, number when carrying out data filtering or processing
Directly pass through the data in data read and write interface reading database according to filter inserts.Involved system is supported based on pass in this programme
The service logic judgement for being type database (RDBMS) and data query and application based on SQL.It is lost as system has existed history
Residual is already present on relevant database according in traditional Relational DataBase, system needs to inquire when carrying out data filtering
In data or relationship, timing (by the second, point or other times unit, depending on being realized by specific system) or according to event
Data in relevant database are synchronized to NoSQL data-base cluster by triggering；The data being stored in NoSQL data-base cluster
Processing result is also synchronized in relevant database as desired by Data synchronization interface.
The mode of data processing described in this programme as shown in figure 3, application filtering each first according to demand,
Subscribe to some message subject, and the data structure definition of each message subject is as shown in figure 4, include a type (Type), visitor
Family end identifies (ClientID) and semantic topic (Topic) etc., the connection type of each section in each message subject data structure
Including but following methods:
1, it is connected by the character string of special interval character, such as " gateway_dsd2348_relay ", and connects special word
Symbol includes but is not limited to underscore " _ ", asterisk " * ", pound sign " # " etc.；
2, by the length and composition of binary number segment identification each section content, as shown in Figure 5.
In this programme when client will send the data about some semantic topic (Topic), client application journey
Sequence can assemble them into the data structure of system message theme automatically, and the other parts in data structure can be from equipment or visitor
It is read in the attribute of family end, it can also be actively specified by programming.Then, application program by data by system interface A (according to
System implements selected communication protocol) (Pub) is issued to above-mentioned assembled message subject, it is filled by Message Queuing system
It sets and receives, be transmitted to data filtering engine, and be filtered according to message subject by data filtering engine, the mode of filtering is such as
Shown in Fig. 6, filter engine respectively matches (Fig. 6) various pieces of message subject, or carries out OR (logic to each section
Or) conditional combination matching (Fig. 7), or each section progress AND (logical AND) conditional combination is matched (Fig. 8) and various mixed
The combinations matches of conjunction.Match pattern further includes any (full matching) mode, i.e. certain part of message subject or multiple portions is to appoint
Meaning value is all by successful match.According to different matching results, data filtering engine executes corresponding data processor (as schemed
6, Fig. 7 and Fig. 8), the executive mode of data processor process includes but local function call, RPC (remote process tune
With), REST API and Webservice etc..
Data processor carries out condition matching and secondary filter according to the TAG for including in each data, as shown in figure 9,
Every pending data all includes one group of TAG value in system involved in the present invention, and the overall structure of data can be one
JSON format and XML format, naturally it is also possible to be other data representation formats；And TAG value can for number, character string or other
Data structure；When data processor is filtered matching to TAG value, single TAG can both be matched, it can also be into
The matching of the combination conditions such as row OR or AND.
In system involved in the present invention, database or forwarding/road can be written into for the data after routine processes
By giving other processing routines or client application, if desired forward, it is right to one or more which will be published (Pub)
The message subject (as shown by dotted lines in figure 3) answered, and the client application by having subscribed corresponding theme or other processing routines
It receives.Whether data that treated, which need to forward, following two deciding means:
1), the mark in need being forwarded in the TAG that data include, such as " relay_to_ "；
2) it, is determined by the logic of data processor, such as certain type of data need to forward, or treated data are full
Certain condition (such as larger than some threshold value) of foot etc..
Whether the system involved in this programme needs data database is written, be determined by data processor,
And the foundation being written includes following two situation:
1) mark in need for being stored in database in the TAG that data include, such as " DB "；
2) it is determined by the logic of data processor, as when data generate the data in Mr. Yu's section time needed that data are written
Library, and the data of other times section do not need write-in etc..
The foregoing is merely presently preferred embodiments of the present invention, is not intended to limit the invention, it is all in spirit of the invention and
Within principle, any modification, equivalent replacement, improvement and so on be should all be included in the protection scope of the present invention.
Claims (8)
1. real time data processing and storage system, the system is the message transmission mode based on Pub/Sub, and data are acquired
End, application end, processing end, storage end and displaying end are attached by way of loose coupling, interactively with each other, are formed towards clothes
The system architecture of business, it is characterised in that: the system includes Message Queuing Services module, application program module, data base set
System module, real time data display module and data filter engine module, the Message Queuing Services module are a number in real time
According to channel agent, realize that based on communication protocol include AMQP, MQTT, Websocket and TCP, the Message Queuing Services
Module provides pipeline for real-time Data Transmission；
The application program module realizes the input or output of data, the master of application program module operation by system A interface
Body includes personal computer, server, mobile terminal；
The real time data display module is a page program based on Web, is assisted by HTTP, Websocket or MQ
View obtains data, and real-time exhibition from Message Queuing Services device；
The Database Systems module includes system R and NoSQL database；
The data filtering engine modules are a shell script performing environment, and the program of the data filtering engine modules is transported
Computer language when row includes Javascript, Java, Perl, Python, Scala and Erlang；
The system also includes the data processing card module of plug-in, the card module is directly acted in transmission channel
The data of middle flowing, the card module are submitted to data filtering engine modules and are run by way of Dynamical Deployment；If being
System needs already present data in dependence system to carry out logic judgment when carrying out data filtering or processing, and data filtering is inserted
Part directly passes through the data in data read and write interface reading database；
The data processing engine module subscribes to corresponding message subject according to demand；When client will be sent about some language
When the data of adopted theme, client application can be assembled into the data structure of system message theme automatically, and message master
Other parts in the data structure of topic can be read from equipment or client terminal attribute, can also actively be referred to by programming
It is fixed；Then, data are published to above-mentioned assembled message subject by system interface by application program, are filled by Message Queuing system
It sets and receives, be transmitted to data filtering engine, and be filtered according to message subject by data filtering engine；Filter engine difference
The various pieces of message subject are matched；According to different matching results, data filtering engine executes corresponding data
Processing routine；
The data filtering engine modules are equipped with data processor；Pending data includes one in the data processor
Group TAG value, the data processor carry out condition matching and secondary filter according to the TAG for including in each data；
Database can be written into for the data after routine processes, or forward/be routed to other processing routines or client
Application program is held, is if desired forwarded, which will be distributed to one or more corresponding message subjects, and corresponding by having subscribed
The client application of theme or other processing routines receive；Whether data are needed database is written, by data
Program is managed to determine.
2. real time data processing according to claim 1 and storage system, it is characterised in that: card module is program source
Program file after code or compiling.
3. real time data processing according to claim 1 and storage system, it is characterised in that: the number of the message subject
It include a type (Type), client identification (ClientID) and semantic topic (Topic) according to structure.
4. real time data processing according to claim 3 and storage system, it is characterised in that: the number of the message subject
Connection type according to each section in structure includes:
4-1, it is connected by the character string of special interval character, the special interval character includes underscore " _ ", asterisk " * "
With pound sign " # "；
4-2, pass through the length and composition of binary number segment identification each section content.
5. real time data processing according to claim 1 and storage system, it is characterised in that: the data processing journey
The executive mode of sequence process includes local function call, RPC, REST API or Webservice.
6. real time data processing according to claim 5 and storage system, it is characterised in that: the overall structure packet of data
JSON format or XML format are included, the TAG value includes number or character string；The data processor carries out TAG value
When filtering matching, including single TAG is matched, and carries out the matching of OR AND combination condition.
7. real time data processing according to claim 6 and storage system, it is characterised in that: the data processor
Data after processing need to forward when there is following situations:
The mark in need being forwarded in the TAG that 7-1, data include；
7-2, it is determined by the logic of data processor, comprising: certain type of data need to forward, or treated data are full
The corresponding condition of foot.
8. real time data processing according to claim 6 and storage system, it is characterised in that: the data processor
Data after processing need that database is written when there is following situations:
The mark in need for being stored in database in the TAG that 8-1, data include；
8-2, it is determined by the logic of data processor, comprising: when data generate the data in Mr. Yu's section time need that number is written
It does not need to be written according to the data in library, and other times section.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610281670.4A CN105956082B (en) | 2016-04-29 | 2016-04-29 | Real time data processing and storage system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610281670.4A CN105956082B (en) | 2016-04-29 | 2016-04-29 | Real time data processing and storage system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN105956082A CN105956082A (en) | 2016-09-21 |
| CN105956082B true CN105956082B (en) | 2019-07-02 |
Family
ID=56913056
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610281670.4A Active CN105956082B (en) | 2016-04-29 | 2016-04-29 | Real time data processing and storage system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN105956082B (en) |
Families Citing this family (19)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107959699A (en) * | 2016-10-18 | 2018-04-24 | 中车株洲电力机车研究所有限公司 | A kind of WebSocket data delivery systems and method based on Stream Processing |
| CN108614820B (en) * | 2016-12-09 | 2021-01-15 | 腾讯科技（深圳）有限公司 | Method and device for realizing streaming source data analysis |
| CN108255802B (en) * | 2016-12-29 | 2021-08-24 | 北京国双科技有限公司 | Universal text parsing architecture and method and device for parsing text based on architecture |
| US11323519B2 (en) * | 2017-04-19 | 2022-05-03 | Microsoft Technology Licensing, Llc | Internet of things pub-sub data publisher |
| CN107241422B (en) * | 2017-06-23 | 2020-08-11 | 浪潮云信息技术股份公司 | Method for synchronizing external user and user group information into Apache Range in real time |
| CN107480283A (en) * | 2017-08-23 | 2017-12-15 | 九次方大数据信息集团有限公司 | Realize the method, apparatus and storage system of big data quick storage |
| CN107766538A (en) * | 2017-10-28 | 2018-03-06 | 杭州安恒信息技术有限公司 | Data filtering processing module and synchronous, asynchronous filter method based on java |
| CN108268329B (en) * | 2017-12-28 | 2022-01-28 | 武汉长江仪器自动化研究所有限公司 | Data processing method and device for grouting recorder network terminal |
| CN108304267B (en) * | 2018-01-31 | 2022-03-22 | 中科边缘智慧信息科技（苏州）有限公司 | Multi-source data leading method with high reliability and low resource overhead |
| CN108133062A (en) * | 2018-02-02 | 2018-06-08 | 北京火花日志科技有限公司 | A kind of data processing system |
| CN109684416A (en) * | 2018-11-13 | 2019-04-26 | 国电南京自动化股份有限公司 | A kind of high concurrent real-time history data storage system |
| CN109831385B (en) * | 2019-01-22 | 2021-11-05 | 北京奇艺世纪科技有限公司 | Message processing method and device and electronic equipment |
| CN110109996A (en) * | 2019-05-15 | 2019-08-09 | 浪潮通用软件有限公司 | A kind of industry internet equipment real-time data acquisition and storage method and system |
| CN112995263B (en) * | 2019-12-18 | 2022-11-22 | 中国移动通信集团陕西有限公司 | Network priority data processing system |
| CN111428128B (en) * | 2020-02-20 | 2021-09-10 | 北京达佳互联信息技术有限公司 | Method for generating pluggable recommendation system, service recommendation method, device and equipment |
| CN111309335B (en) * | 2020-02-28 | 2023-08-15 | 腾讯音乐娱乐科技（深圳）有限公司 | Compiling method and device of plug-in application and computer readable storage medium |
| CN112073506B (en) * | 2020-09-04 | 2022-10-25 | 哈尔滨工业大学 | IPv6 and message queue-based complex electromagnetic data acquisition method |
| CN112422613B (en) * | 2020-09-16 | 2022-02-01 | 北京中兵数字科技集团有限公司 | Data processing method, data processing platform and computer readable storage medium |
| CN113220720B (en) * | 2021-06-08 | 2022-11-08 | 上海天旦网络科技发展有限公司 | Real-time second-level data processing and query method and system |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1959717A (en) * | 2006-10-09 | 2007-05-09 | 北京道达天际软件技术有限公司 | System and method for preprocessing mass remote sensing data collection driven by order form |
| CN103164721A (en) * | 2013-03-15 | 2013-06-19 | 安徽皖通科技股份有限公司 | Car connected network side information data collection and processing method |
| CN104144071A (en) * | 2013-05-10 | 2014-11-12 | 北京新媒传信科技有限公司 | System log processing method and platform |
| CN104572975A (en) * | 2014-12-31 | 2015-04-29 | 中国石油天然气股份有限公司 | Real-time data processing analysis system |
| CN105512939A (en) * | 2015-12-04 | 2016-04-20 | 中国建设银行股份有限公司 | Foreign exchange transaction-related data storage and query method, declaration method and system |
-
2016
- 2016-04-29 CN CN201610281670.4A patent/CN105956082B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1959717A (en) * | 2006-10-09 | 2007-05-09 | 北京道达天际软件技术有限公司 | System and method for preprocessing mass remote sensing data collection driven by order form |
| CN103164721A (en) * | 2013-03-15 | 2013-06-19 | 安徽皖通科技股份有限公司 | Car connected network side information data collection and processing method |
| CN104144071A (en) * | 2013-05-10 | 2014-11-12 | 北京新媒传信科技有限公司 | System log processing method and platform |
| CN104572975A (en) * | 2014-12-31 | 2015-04-29 | 中国石油天然气股份有限公司 | Real-time data processing analysis system |
| CN105512939A (en) * | 2015-12-04 | 2016-04-20 | 中国建设银行股份有限公司 | Foreign exchange transaction-related data storage and query method, declaration method and system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN105956082A (en) | 2016-09-21 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN105956082B (en) | Real time data processing and storage system | |
| CN110784419B (en) | Method and system for visualizing professional railway electric service data | |
| CN106547766A (en) | A kind of data access method and device | |
| CN104182389B (en) | A kind of big data analyzing business intelligence service system based on semanteme | |
| CN105912559B (en) | Complicated entity and relationship are extracted from unstructured data | |
| CN109034993A (en) | Account checking method, equipment, system and computer readable storage medium | |
| CN110399550A (en) | A kind of information recommendation method and device | |
| CN109684370A (en) | Daily record data processing method, system, equipment and storage medium | |
| CN109492012A (en) | A kind of method, apparatus and storage medium of data real-time statistics and retrieval | |
| CN110019267A (en) | A kind of metadata updates method, apparatus, system, electronic equipment and storage medium | |
| US20210385251A1 (en) | System and methods for integrating datasets and automating transformation workflows using a distributed computational graph | |
| CN105993011A (en) | Pattern matching across multiple input data streams | |
| CN106022708A (en) | Method for predicting employee resignation | |
| CN108681569A (en) | A kind of automatic data analysis system and its method | |
| CN110300084B (en) | IP address-based portrait method and apparatus, electronic device, and readable medium | |
| CN103248677B (en) | The Internet behavioural analysis system and method for work thereof | |
| CN110795697B (en) | Method and device for acquiring logic expression, storage medium and electronic device | |
| CN110019087A (en) | Data processing method and its system | |
| CN107256206A (en) | The method and apparatus of character stream format conversion | |
| CN110196790A (en) | The method and apparatus of abnormal monitoring | |
| CN106534241A (en) | Real-time data updating method, device and system | |
| CN109710413A (en) | A kind of integral Calculation Method of the rule engine system of semi-structured text data | |
| JP2014002482A (en) | Influence prediction apparatus, influence prediction method and program | |
| CN104199889A (en) | RTLogic big data processing system and method based on CEP technology | |
| CN110851136A (en) | Data acquisition method and device, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB02 | Change of applicant information |
Address after: 518000 B502, light Science Park, No. 7 West West Road, Shenzhen, Guangdong, Nanshan District Applicant after: Shenzhen great number of Technology Co., Ltd. Address before: 518000 B502, light Science Park, No. 7 West West Road, Shenzhen, Guangdong, Nanshan District Applicant before: SHENZHEN QIANHAI DASHUDIAN TECHNOLOGY CO., LTD. |
|
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20210108 Address after: 518000 Guangdong city of Shenzhen province Nanshan District South Road No. 10 four Patentee after: Liu Fujianbo Address before: 518000 B502, Languang Science Park, No.7, Xinxi Road, Nanshan District, Shenzhen City, Guangdong Province Patentee before: Shenzhen great number of Technology Co.,Ltd. |
|
| TR01 | Transfer of patent right | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20210331 Address after: 518000 f3.8-8b4 (b), Tianjing building, Tian'an chegongmiao Industrial Zone, Shatou street, Futian District, Shenzhen City, Guangdong Province Patentee after: SHENZHEN ZHIYUAN YOUXUE EDUCATION TECHNOLOGY Co.,Ltd. Address before: 518000 Guangdong city of Shenzhen province Nanshan District South Road No. 10 four Patentee before: Liu Fujianbo |