CN102081641A - Design of GPU-based real-time data stream high-speed matching analysis system - Google Patents
Design of GPU-based real-time data stream high-speed matching analysis system Download PDFInfo
- Publication number
- CN102081641A CN102081641A CN 201010508723 CN201010508723A CN102081641A CN 102081641 A CN102081641 A CN 102081641A CN 201010508723 CN201010508723 CN 201010508723 CN 201010508723 A CN201010508723 A CN 201010508723A CN 102081641 A CN102081641 A CN 102081641A
- Authority
- CN
- China
- Prior art keywords
- data stream
- data
- matching analysis
- time data
- real time
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Abstract
The invention provides a design of a graphics processing unit (GPU)-based real-time data stream high-speed matching analysis system, and particularly makes a contribution to improvement on the efficiency of real-time data stream high-speed matching analysis by utilizing the GPU. The system at least comprises the following parts, namely a set of data stream receiving device, a set of mass data stage storage device, a set of complete real-time data stream high-speed matching analysis system and a set of mobile intelligent terminal, wherein the data stream receiving device receives monitoring data and stores the data into a distributed workstation; the mass data stage storage device is used for classified normalization and stage storage of the data on the workstation and, when a queried real-time data matching analysis request is received, the system checks whether a user is authorized, confirms and performs matching analysis processing in the real-time data stream high-speed matching analysis system, and wireless communication hardware transmits an analyzed result to a portable terminal when the wireless communication hardware reaches an agreement with a corresponding agent. The system can effectively monitor and analyze mass data streams on a portable device, has higher analysis efficiency, and is particularly suitable for engineering personnel going out to work.
Description
The invention provides the design of a kind of real time data stream high speed The matching analysis system, especially utilize GPU to improve the design of real time data stream matching inquiry speed.
Background technology
Real time data stream detects and is stored as the bottleneck of the business that detects for data stream, how to design a kind of brand-new mass data stage memory storage, this apparatus features is the data in the distributed work station regularly can be carried out classified finishing, and the staff is needed the real time data stream quick calling of match query.And expired data stream stores into independently and prepares against the function of doing long term data excavation rational analysis in the massage storage in the memory mechanism.
The monitoring means of real time data stream is brought in constant renewal in and to the continuous demand of result data degree of accuracy, make to real time data stream and national standard accurately, The matching analysis becomes the professional problem that needs solution of current real time data stream monitoring fast.Compare with traditional data structure, characteristics such as it is real-time, continuous, endless that data stream has, these singularity of data stream have brought new opportunities and challenges for the research of the information processing technology and application. and the characteristics of data stream have determined that its main match query mode is the continuous-query coupling.Continuous-query is meant execution continuously for a long time after an inquiry is established, along with the arrival of new data will constantly produce new match query result.Because data stream is the set of series of discrete state, if the data base querying matching way that adopts tradition to calculate based on CPU, be difficult to obtain fast match query result accurately under a lot of querying conditions, because traditional data stream storage mode all deposits mass data flow in storage mediums such as disk, when carrying out the match query order, need carry out a large amount of I/O operations, its response speed is slow, is difficult to adapt to the requirement of real-time match query.In fact the data of latest stage are more important than historical data, so the data of the latest stage often that people are concerned about in the data stream The matching analysis.
Summary of the invention
The invention provides the design of a kind of real time data stream high speed The matching analysis system, particularly, can be used on the portable terminal real time data stream being carried out the high speed The matching analysis based on the design of the real time data stream high speed The matching analysis system of GPU.
Native system mainly comprises the data stream receiving trap, mass data stage memory storage, the complete real time data stream high speed The matching analysis system of a cover, wireless telecommunications system, and mobile intelligent terminal; Implementation procedure is such, observation satellite is passed to ground data stream receiving station with observed result as data stream, the station spreads Monitoring Data to data stream receiving station, the data stream receiving trap of these two kinds of forms has received Monitoring Data stream, and data stream is stored into the workstation of distribution various places, mass data stage memory storage is responsible for the regular and stage storage the data qualification on the workstation, afterwards when receiving the The matching analysis request of mobile intelligent terminal, system checks at first whether the user is authorized to, confirming will carry out from the system of real time data stream high speed The matching analysis after errorless The matching analysis handles, after The matching analysis result is responsible for by wireless telecommunications system and corresponding commission merchant reaches an agreement on, send on the mobile intelligent terminal equipment.The present invention can carry out The matching analysis to real time data stream effectively on portable unit, The matching analysis speed is very fast, accuracy rate is higher, the engineering staff of the operation that is particularly suitable for going out.
Description of drawings
Fig. 1 is the synoptic diagram of disclosed real time data stream high speed The matching analysis system.
Fig. 2 is the inquiry frame diagram of disclosed real time data stream high speed The matching analysis system.
Fig. 3 is the querying flow figure of disclosed real time data stream high speed The matching analysis system.
Fig. 4 is the inside configuration synoptic diagram of disclosed real time data stream high speed The matching analysis system.
Embodiment
Technological difficulties to be solved by this invention be how satisfy to the real time data matching inquiry provide at a high speed, the data stream The matching analysis mechanism of pinpoint accuracy, to tackle the application demand of various data streaming service information.
As Fig. 1, provide the real time data flow receiving device to comprise various different classes of monitoring devices, detect equipment such as data sink, workstation.Monitoring device has observation satellite, the station etc., has different monitoring equipments towards different field; Each monitoring equipment all has corresponding with it data stream receiving trap, detects DRS, station DRS etc. such as satellite.These devices provide the most original measured data for us, these data can be according to instruction storage in distributed workstation, the data of this moment are not through unified data preprocessing process, because data volume is huge, so will take a large amount of stand-by period of user when carrying out the The matching analysis instruction, be not suitable for outworker's query calls at any time.Improved method adopts mass data stage memory storage and based on the real time data stream high speed The matching analysis system of GPU, to storing and computing velocity all is once tremendous improvement and optimization.
Mass data stage memory storage is used for preserving being stored in the distributed data base of data stream that each workstation sends and stageization.This is comprising the pre-service to raw data, data qualification, the operation of warehouse-in.The data pre-service is responsible for the data that transmit are carried out operations such as data cleansing, denoising, and the purpose in this step is to reduce data redundancy, data inconsistency, avoids reading invalid dirty data.Data qualification is that raw data is classified by attribute information and format differences, is different classes of data such as station data and satellite monitoring data, must be stored in the different databases.Data qualification finishes, and next is exactly these data are carried out in-stockroom operation.
Real time data stream high speed The matching analysis system based on GPU mainly comprises intermediary device, matched-field processors.The query requests of system receiving terminal is obtained data from heterogeneous database.Query Result must just can turn back on the portable terminal by wireless telecommunications hardware and the agential transfer of wireless telecommunications.The wireless telecommunications hardware is here realized Query Result is passed to the commission merchant, is responsible for transferring electromagnetic signal to digital signal by the commission merchant, sends to terminal request.
As Fig. 2, the real time data stream high speed The matching analysis system based on GPU among the present invention mainly is made up of intermediary device, matched-field processors.Intermediary device receives user's query requests, is responsible for local heterogeneous data source mutual by intermediary device.System offers global schema of user, and the data of mass data stage memory storage are carried out matching inquiry.Matched-field processors is carried out query requests, and the result is returned to intermediary device, and intermediary device carries out after the overall treatment result being returned to the user.Wherein intermediary device is made up of integrated manager and query processor, and bookkeepings such as the responsible increase to distribution local data of integrated manager, deletion can easily be managed each data source by global map, realize the data integration management; Be responsible for local data simultaneously and arrive the integrated of global data.Matched-field processors is responsible for accepting user's query requests, and Query Result is returned.
Continuing referring to Fig. 3, is the system handles process flow diagram to the data continuous query, mainly comprises following steps: at first open the web browser in the terminal (following is example with the mobile phone), open corresponding website; Then carry out query manipulation; System checks earlier whether this user has authority, if not then getting back to login interface; If this user has authority, then enter real time data stream high speed The matching analysis system and carry out query manipulation, system returns to terminal with Query Result, finishes.
Continue referring to Fig. 4, this figure is based on the core processing unit figure of the data stream matched-field processors of GPU among Fig. 2.This module is the nucleus module of system, is based on GPU and carries out high-performance calculation, is divided into data stream Uniform Scaling processing, Normalization processing and three steps of data stream matching treatment.The data stream of in calculator memory data stream being carried out that UniformScaling handles, Normalization handles and will change is sent in the GPU video memory, the video memory of GPU uses Scaling Subsequence Matrix memorying data flow and utilizes the moving window model and standardized data stream carries out the high speed The matching analysis, after at last Query Result being gathered, feed back to the user., because the characteristics of GPU are the intensive computings at a large amount of data of the same type, so the advantage that the numerical evaluation of matrix-type is operated makes the matched data flow operation more quick, matching precision is higher.
In sum, the present invention is based on the design of the real time data stream high speed The matching analysis system of GPU, Monitoring Data by various monitoring device aggregated data stream types, and through storing in the mass data stage memory storage after the data pre-service, portable terminal can be carried out corresponding real time data stream matching request by authority at any time.
The above only is an illustrative, but not is restricted.This description should not be construed as limitation of the present invention, and any spirit of the present invention and category of not breaking away from all should be contained within the application range.
Claims (8)
1. based on the design of the real time data stream high speed The matching analysis system of GPU, it is characterized in that, should include:
One sets of data flow receiving device is used for receiving magnanimity satellite monitoring data and station data by the monitoring gained;
One cover mass data stage memory storage, this device is connected with distributed workstation, is responsible for regularly with workstation data classified finishing, and the stage storage;
One cover real time data stream high speed The matching analysis system for the The matching analysis request that receives mobile intelligent terminal, mates with normal data numerical value from mass data stage memory storage, and whether decision returns analysis result according to authorization conditions at last;
One cover wireless telecommunications system, the analysis result that real time data stream high speed The matching analysis system produces need be transferred on the portable equipment by the wireless telecommunications equipment, and wireless telecommunications hardware plays the effect of signal transfer;
One cover mobile intelligent terminal, but portable set.
2. data stream receiving trap according to claim 1 is characterized in that this data stream receiving trap is by the station, satellite monitoring multi-faceted monitoring and the gathering-device formed of receiving trap such as receiving trap as a result.
3. mass data stage memory storage according to claim 1 is characterized in that this mass data stage memory storage receives up-to-date data stream with the storage of stage from each workstation.
4. real time data stream high speed The matching analysis according to claim 1 system is characterized in that isomery geodata inquiry system comprises with the lower part to form:
One cover intermediary device module, analysis module is called and passed to reception user's query requests to the data in the fixedly duration before the current time with the form of data stream, and last intermediary device carries out after the overall treatment result being returned to the user at analysis module.
One cover matched-field processors module responsiblely will be utilized the powerful computation capability of GPU to carry out The matching analysis by the data query national standard corresponding with it in real time, analyzes the degree of its similar coupling, and returns to intermediary device after the result changed.
5. real time data stream high speed The matching analysis according to claim 4 system is characterized in that intermediary device is made up of request receiving management device and matched-field processors two parts.
6. real time data stream high speed The matching analysis according to claim 4 system is characterized in that the matched-field processors module carries out high speed analysis with two data stream and calculate coupling, and execution analysis format conversion as a result.
7. wireless telecommunications hardware according to claim 1 is characterized in that the query analysis result is changed into the form that is easy to wireless transmission, can transfer electromagnetic signal to by the commission merchant.
8. but portable set according to claim 1 is characterized in that it is movably, smart mobile phone or PDA.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN 201010508723 CN102081641A (en) | 2010-10-15 | 2010-10-15 | Design of GPU-based real-time data stream high-speed matching analysis system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN 201010508723 CN102081641A (en) | 2010-10-15 | 2010-10-15 | Design of GPU-based real-time data stream high-speed matching analysis system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN102081641A true CN102081641A (en) | 2011-06-01 |
Family
ID=44087605
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN 201010508723 Pending CN102081641A (en) | 2010-10-15 | 2010-10-15 | Design of GPU-based real-time data stream high-speed matching analysis system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN102081641A (en) |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103780660A (en) * | 2012-10-25 | 2014-05-07 | 中国石油化工股份有限公司 | GPU cluster monitoring system and method for issuing monitoring alarm |
| CN108037959A (en) * | 2017-11-30 | 2018-05-15 | 努比亚技术有限公司 | Interface intelligence merging method, mobile terminal and computer-readable recording medium |
| CN109086397A (en) * | 2018-07-28 | 2018-12-25 | 重庆柚瓣家科技有限公司 | Dynamic report generation method based on inquiry dimension |
| CN110019501A (en) * | 2017-08-24 | 2019-07-16 | 深圳市金证科技股份有限公司 | A kind of collecting method, device and terminal device |
| CN110108938A (en) * | 2019-04-17 | 2019-08-09 | 北京航空航天大学 | A kind of real-time electromagnetic spectrum analysis system and method based on GPU data flow driven |
| CN111914031A (en) * | 2020-08-11 | 2020-11-10 | 中国科学院地质与地球物理研究所 | Satellite-borne magnetic field data processing system based on big data |
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101582090A (en) * | 2009-07-07 | 2009-11-18 | 北京黑米世纪信息技术有限公司 | Distributed processing method and system based on WEB analysis |
-
2010
- 2010-10-15 CN CN 201010508723 patent/CN102081641A/en active Pending
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101582090A (en) * | 2009-07-07 | 2009-11-18 | 北京黑米世纪信息技术有限公司 | Distributed processing method and system based on WEB analysis |
Cited By (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103780660A (en) * | 2012-10-25 | 2014-05-07 | 中国石油化工股份有限公司 | GPU cluster monitoring system and method for issuing monitoring alarm |
| CN110019501A (en) * | 2017-08-24 | 2019-07-16 | 深圳市金证科技股份有限公司 | A kind of collecting method, device and terminal device |
| CN108037959A (en) * | 2017-11-30 | 2018-05-15 | 努比亚技术有限公司 | Interface intelligence merging method, mobile terminal and computer-readable recording medium |
| CN108037959B (en) * | 2017-11-30 | 2021-01-01 | 努比亚技术有限公司 | Intelligent interface merging method, mobile terminal and computer readable storage medium |
| CN109086397A (en) * | 2018-07-28 | 2018-12-25 | 重庆柚瓣家科技有限公司 | Dynamic report generation method based on inquiry dimension |
| CN110108938A (en) * | 2019-04-17 | 2019-08-09 | 北京航空航天大学 | A kind of real-time electromagnetic spectrum analysis system and method based on GPU data flow driven |
| CN110108938B (en) * | 2019-04-17 | 2020-07-10 | 北京航空航天大学 | Real-time electromagnetic spectrum analysis system and method based on GPU data stream driving |
| CN111914031A (en) * | 2020-08-11 | 2020-11-10 | 中国科学院地质与地球物理研究所 | Satellite-borne magnetic field data processing system based on big data |
| CN111914031B (en) * | 2020-08-11 | 2021-03-16 | 中国科学院地质与地球物理研究所 | Satellite-borne magnetic field data processing system based on big data |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110460656B (en) | Industry environmental protection thing networking remote monitoring cloud platform | |
| CN109271412A (en) | The real-time streaming data processing method and system of smart city | |
| CN102081641A (en) | Design of GPU-based real-time data stream high-speed matching analysis system | |
| CN105608758A (en) | Big data analysis platform apparatus and method based on algorithm configuration and distributed stream computing | |
| CN104077301A (en) | Housekeeping information comprehensive service platform based on cloud computing mobile positioning off-line service | |
| CN109189835A (en) | The method and apparatus of the wide table of data are generated in real time | |
| CN110019267A (en) | A kind of metadata updates method, apparatus, system, electronic equipment and storage medium | |
| CN110019087A (en) | Data processing method and its system | |
| CN108228743A (en) | A kind of real-time big data search engine system | |
| CN109581914A (en) | A kind of Intelligent internet of things service platform Products Eco system | |
| CN106873945A (en) | Data processing architecture and data processing method based on batch processing and Stream Processing | |
| CN107729399A (en) | The method and apparatus of data processing | |
| CN104486116A (en) | Multidimensional query method and multidimensional query system of flow data | |
| CN108494090A (en) | A kind of energy net connection intelligent apparatus and system based on block chain | |
| CN110932405A (en) | Intelligent monitoring and analyzing system for power transformation equipment based on big data | |
| CN105405071A (en) | Method for constructing unified data model for enterprise | |
| CN103900534A (en) | Island resource dynamic monitoring system based on 3S technology | |
| CN105515892A (en) | Mobile terminal data sending monitoring device and application method thereof | |
| Swaroop et al. | Mobile distributed real time database systems: A research challenges | |
| CN103900533A (en) | Island resource dynamic monitoring method based on 3S technology | |
| CN103383649A (en) | Design model of complex virtual instrument system | |
| CN110378641A (en) | A kind of method and apparatus counting unmanned vehicle information | |
| CN109165203A (en) | Large public building energy consumption data based on Hadoop framework stores analysis method | |
| Yu | Research of 4th generation GIS software | |
| CN103746994A (en) | Data acquisition method for mobile intelligent terminal and high-level application system thereof |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C02 | Deemed withdrawal of patent application after publication (patent law 2001) | ||
| WD01 | Invention patent application deemed withdrawn after publication |
Application publication date: 20110601 |