CN105760449B - A kind of cloud method for pushing towards multi-source heterogeneous data - Google Patents
A kind of cloud method for pushing towards multi-source heterogeneous data Download PDFInfo
- Publication number
- CN105760449B CN105760449B CN201610077551.7A CN201610077551A CN105760449B CN 105760449 B CN105760449 B CN 105760449B CN 201610077551 A CN201610077551 A CN 201610077551A CN 105760449 B CN105760449 B CN 105760449B
- Authority
- CN
- China
- Prior art keywords
- data
- push
- cloud
- user
- source heterogeneous
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 50
- 230000008569 process Effects 0.000 claims abstract description 23
- 238000013461 design Methods 0.000 claims abstract description 9
- 239000011159 matrix material Substances 0.000 claims description 59
- 230000005540 biological transmission Effects 0.000 claims description 23
- 238000004891 communication Methods 0.000 claims description 18
- 238000004458 analytical method Methods 0.000 claims description 6
- 238000012986 modification Methods 0.000 claims description 6
- 230000004048 modification Effects 0.000 claims description 6
- 238000006073 displacement reaction Methods 0.000 claims description 5
- 238000000547 structure data Methods 0.000 claims description 4
- 238000013523 data management Methods 0.000 claims description 3
- 239000004615 ingredient Substances 0.000 claims description 3
- 238000002372 labelling Methods 0.000 claims description 3
- 238000007726 management method Methods 0.000 claims description 3
- 238000013507 mapping Methods 0.000 claims description 3
- 238000000926 separation method Methods 0.000 claims description 3
- 239000000126 substance Substances 0.000 claims description 3
- CDDBPMZDDVHXFN-ONEGZZNKSA-N 2-[(e)-3-(1,3-benzodioxol-5-yl)prop-2-enyl]-1-hydroxypiperidine Chemical compound ON1CCCCC1C\C=C\C1=CC=C(OCO2)C2=C1 CDDBPMZDDVHXFN-ONEGZZNKSA-N 0.000 claims 8
- NAWXUBYGYWOOIX-SFHVURJKSA-N (2s)-2-[[4-[2-(2,4-diaminoquinazolin-6-yl)ethyl]benzoyl]amino]-4-methylidenepentanedioic acid Chemical compound C1=CC2=NC(N)=NC(N)=C2C=C1CCC1=CC=C(C(=O)N[C@@H](CC(=C)C(O)=O)C(O)=O)C=C1 NAWXUBYGYWOOIX-SFHVURJKSA-N 0.000 claims 1
- 238000005516 engineering process Methods 0.000 abstract description 7
- 230000014759 maintenance of location Effects 0.000 description 12
- 238000010586 diagram Methods 0.000 description 9
- 238000011161 development Methods 0.000 description 3
- 238000011160 research Methods 0.000 description 3
- 238000012360 testing method Methods 0.000 description 3
- 238000012546 transfer Methods 0.000 description 3
- 241000208340 Araliaceae Species 0.000 description 2
- 235000005035 Panax pseudoginseng ssp. pseudoginseng Nutrition 0.000 description 2
- 235000003140 Panax quinquefolius Nutrition 0.000 description 2
- 230000008901 benefit Effects 0.000 description 2
- 230000015572 biosynthetic process Effects 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 235000008434 ginseng Nutrition 0.000 description 2
- 238000012544 monitoring process Methods 0.000 description 2
- 230000008447 perception Effects 0.000 description 2
- 238000003786 synthesis reaction Methods 0.000 description 2
- LDSJMFGYNFIFRK-UHFFFAOYSA-N 3-azaniumyl-2-hydroxy-4-phenylbutanoate Chemical compound OC(=O)C(O)C(N)CC1=CC=CC=C1 LDSJMFGYNFIFRK-UHFFFAOYSA-N 0.000 description 1
- 241001269238 Data Species 0.000 description 1
- 230000002159 abnormal effect Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000013506 data mapping Methods 0.000 description 1
- 230000007812 deficiency Effects 0.000 description 1
- 238000002716 delivery method Methods 0.000 description 1
- 239000002360 explosive Substances 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 230000002452 interceptive effect Effects 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 230000035484 reaction time Effects 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 230000009897 systematic effect Effects 0.000 description 1
- 238000010200 validation analysis Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/95—Retrieval from the web
- G06F16/953—Querying, e.g. by the use of web search engines
- G06F16/9535—Search customisation based on user profiles and personalisation
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computer And Data Communications (AREA)
- Information Transfer Between Computers (AREA)
Abstract
A kind of cloud method for pushing towards multi-source heterogeneous data, for the feature of multi-source heterogeneous data, the features such as comprehensive mobile interchange software safety and privacy, to calculate the characteristic value and feature vector of multi-source heterogeneous data in distributed environment, for the isomorphism data and isomeric data in quick separating data source, being automatically separated and efficiently pushing for isomorphism data and isomeric data is realized using cloud push technology.The present invention pushes model by design Multidimensional decision-making cloud, cloud push is set to not only reduce the synchrodata amount in data-pushing renewal process, shorten the time, and solves the problems, such as the cross-platform push of multi-source heterogeneous data, it is huge particular for data volume, and work working efficiency under low bandwidth, the insecure situation of network is obviously improved.
Description
Technical field
The present invention relates to a kind of cloud method for pushing towards multi-source heterogeneous data
Background technique
With the rapid development of mobile communication technology, mobile Internet has become the mainstream of internet development, becomes people
Obtain the main channel of information.By mobile Internet, a large amount of multi-source heterogeneous Data Integrations create global letter together
Cease shared environment.In order to meet the needs of users, various mobile applications are in explosive increase, and the data for needing to push are also to multi-source
Change development.These multi-source heterogeneous data propose high requirement to the capture of information on mobile terminal device, are mainly reflected in shifting
Dynamic property and time-bounded two aspects.Mobility requires the message transmission under the conditions of low-power consumption, low rate；Time-bounded require information exists
It is sent on mobile terminal device in stipulated time.
How quickly and effectively to push multi-source heterogeneous data becomes the new problem that society faces.These multi-source heterogeneous data are difficult
It is pushed in shared also ununified method, traditional data push method has not adapted to present user to information yet
Real-time requirement.Current main push mode is that the information of variation is actively sent to user by server, is joined without user
With, reduce interactive number and burden, shorten the reaction time, improve efficiency, but these methods be not all suitable for it is more
The push of source isomeric data.Although IOS and Android platform have the supplying system of oneself, due in network, operating system
With the limitation of application aspect, there is certain limitation in use, Google's cloud messaging service is not available at home yet, iPhone
On APNS be also only applicable to iOS, cross-platform can not push, thus both at home and abroad at present for cross-platform multi-source heterogeneous data also
It can not carry out the push of real-time high-efficiency.
Multi-source heterogeneous data and push mode are studied from different visual angles by domestic and foreign scholars and research institution.It is first
It first, is mainly Yang Wang from the representative sex work of data transfer direction, Bharadwaj Veeravall devises one
Kind obtains the algorithm of cloud sharing data stage by stage, can effectively control the transmission cost of data.Domestic Xu Fulong, Liu Ming
Et al. further provide it is a kind of based on relative distance perception dynamic data transmission strategy, using sensor node to convergent point
Relative distance carry out the size of calculate node transmission probability, and while transmitting in this, as message, selects the foundation of next-hop.
There are Yang Fang-Chun, Su Sen to propose from the research in decision direction a kind of based on Fuzzy Multiple Attribute Decision Making
The Optimization Selection Algorithm of theoretical semantic Web service combination, which can evaluate is retouched with real number, interval number and language type data
The information stated, to carry out integrated decision-making.The Jiang Qianyue of Tongji University, Zhang Yaying further provide a kind of based on fuzzy synthesis
The server push delivery method of decision, this method combine traditional long poll technology and poll, have obtained a kind of combined type wheel
Inquiry technology.
Realizing in systematic research that the Liu Xin of the Institute of Software, Chinese Academy of Science, Chen Wei are proposed using push technology
A kind of web tree component based on AJAX and Server Push is provided for user and is similar in windows explorer
To the basic function and user experience of catalogue tree operations.
But above method is realized and is pushed to data mapping simply by modification push mode, and there is no excessive
Consider the push problem of multi-source heterogeneous data.
Summary of the invention
In order to overcome the problems, such as that existing method for pushing can not solve the push of multi-source heterogeneous data, the present invention proposes a kind of face
To the cloud method for pushing of multi-source heterogeneous data, this method is directed to the feature of multi-source heterogeneous data, comprehensive mobile interchange software safety
And the features such as privacy, is used for quick separating to calculate the characteristic value and feature vector of multi-source heterogeneous data in distributed environment
Isomorphism data and isomeric data in data source realize being automatically separated for isomorphism data and isomeric data using cloud push technology
With efficient push.
In order to solve the above-mentioned technical problem it provides the following technical solutions.
A kind of cloud method for pushing towards multi-source heterogeneous data, includes the following steps：
The first step：It designs cloud and pushes platform, process is as follows：
1.1 clouds push the message of platform push three types：Notice, transparent transmission message and Rich Media notify as to mobile terminal
Push is presented in the notification message of system notification bar；Transparent transmission message, which refers to, sends visitor for customized content in a manner of transparent transmission
Family end, developer can set rule in client in advance, carry out message customization；Rich Media refer to push picture, video,
The information of the forms such as audio, network address；
1.2 clouds push platform should be supported to disappear to all users or according to labeling to the push of specific user group simultaneously
User grouping is given grouping user different labels by breath according to the message content that user subscribes to, and according to different labels, is carried out
The push of corresponding contents carries out single label to single user, is finally pushed according to label substance；
1.3 clouds push platform and provide user information and notification message statistical information, are counted and are believed according to the feedback information of user
Cease retention ratio, communication rate, flow consumption etc.；
1.4 clouds push platform cross-platform can use (PC, iOS, Android), and user can also be according to their own needs
Add custom feature；
Second step：The acquisition of multi-source heterogeneous data, process are as follows：
Multi-source heterogeneous data in environment not in homologous ray are considered as cloud data by 2.1, the different weights given according to system,
It will be different modules, such as notification module, Rich Media's module, transparent transmission message module by cloud data classification；
2.2 clouds push platform is managed according to the module information that user subscribes to, will according to the module that different user is subscribed to
User is grouped, and the user with equal modules is with group and labelled.The information of registration user is managed simultaneously；
2.3 cloud push server are managed cloud data information, and information is managed according to the time, record every time
The data of update, use when to push in next step；
2.4 clouds push platform pre-processes the more new information in (2.3), and data format is identical, and weight is identical
It is divided into different grouping, facilitates the separation of isomorphism data and isomeric data.The present invention proposes MDCP (Multiple Decision
Cloud Push) model is by the data of access to information database, by determining that weight and attribute separation carry out decision push, as entirely
" subscription-collection-decision-push " period；
Third step：Information collect after the completion of, need MDCP model to separate cloud data, be divided into multi-source heterogeneous data and
Multi-source isomorphism data, process are as follows：
3.1 multi-source heterogeneous data component analyses
Due to being wanting in consideration to isomerism problem, lead to the reduction of the efficiency in push, or even can not work.So in face
When to isomeric data, a suitable push mode is selected to be necessary.
Problem to be solved of the present invention is exactly the cloud data that are distributed on different Cloud Servers using these to be counted
According to principal component, pass through each key data signal component value XiSubtract each other the deviant in this direction as the ingredient with the μ provided,
By the main component value X of each datai(i=1,2...n) it inserts in matrix S.
S=[x1,x2,x3...xn] (1)
It sums after the diagonal matrix ST by matrix S multiplied by it and is averaged again again, since the Matrix Multiplication is with its diagonal matrix
It obtains being determining numerical value afterwards, so the value can be used to indicate whole isomery degree, be indicated with V.
Wherein μ is the weighted value obtained according to assessment, obtains its value range according to different systems, and N is sample total,
Since V is determining numerical value at this time, the isomery degree available feature value V of the sample is indicated.
It needs to push if there is multi-source heterogeneous data, the principal component of each data cannot be divided into X at this time1~XN, need at this time
Want XiBecome Xij, so matrix S becomes：
At this point, V is matrix, it is considered as eigenmatrix, so diagonal matrix of the V multiplied by it, can release the feature of cloud data
Value k：
|V-kE| (4)
V gives feature vector, has found out corresponding characteristic value k, the size and V common metrics of characteristic value k by abbreviation
The isomery degree of different cloud data.
But the problem of data transfer load is not accounted in this method, different cloud data transmission in distributed environment
By communication rate, the limitation of the factors such as computing capability, it is desirable to reduce the data traffic between Cloud Server, avoid directly transmitting
Data can be carried out in a matrix fashion permutation and combination, then carry out cloud push based on this data structure by great amount of samples.
The push of 3.2 multi-source isomorphism data, process are as follows：
In cloud data, although there is a large amount of multi-source heterogeneous data, there is also magnanimity multi-source isomorphism data,
When the push content that user needs belongs to isomorphism data, data are pressed into eigendecomposition using covariance matrix, are decomposed same
Structure data can use more efficient push mode, do not need to iterate in each point, saved data communication rates；
Firstly, with sample estimation local variance and mean vector μ in first cloud data station1, then obtain second
Sample estimation local variance and mean vector μ in cloud data station2, until obtaining the sample N of all cloud data stationsμ。Nkμk
For the remaining sample not updated, μKFor sample NKMean vector.In the prescribed limit of local variance and mean vector, then recognize
To be that these cloud data stations provide isomorphism data, average mean vector μ is found out, then pass through NμAnd NkμkFind out average ginseng
Number covariance：
Therefore in the case where each cloud data sample is sufficient, calculate isomorphism data accuracy be it is very big, due to
It is isomorphism data, secondary incremental update equation is:
Wherein (K=1,2,3.....), due in above-mentioned algorithm, it can be noted that covariance matrix be it is symmetrical, because
During push, isomorphism data update every time for this, it is only necessary to transmit half of covariance matrix, MDCP model use is based on
(Message Queuing Telemetry Transport, message queue telemetering transmission, is that IBM is opened to the MQTT of WebSocket
One instant communication protocol of hair) it is pushed, if number of systems is M, the time complexity of number of parameters d, the algorithm are：
O (M, d)=(M-1) (d (d+1)/2+d+1)
The push of 3.3 multi-source heterogeneous data
Multi-source heterogeneous data are extended to isomorphism data main component, in order to facilitate mathematical computations, introduce displacement square
Battle array P, carries out displacement mapping to the sample obtained on Cloud Server, is as a result denoted as y, i.e.,：
Y=(ya,yb)t=PmX (9)
The purpose is to focus on before vector in current sample with identical part in isomorphism data, y is usedaIt indicates,
And different are partially disposed in behind vector, use ybIt indicates, equally, Mean Matrix and covariance matrix is replaced,
As a result it is denoted as μ, ∑ respectively：
μ=(μa,μb)t=Pmμ
μ and permutation matrix are all the last amounts being calculated, and Σ a is the covariance matrix obtained in m sample,
Σ c be from the covariance matrix of elements different in isomorphism data, and Σ b be between them covariance constitute matrix.
The formula meets higher-dimension distribution, so cloud data sample can be calculated using higher-dimension distribution, expression formula is such as
Under, d is the dimension values in higher-dimension distribution:
Y, μ, ∑ is brought by (5) (9) (10) 3 formula to be simplified, and takes natural logrithm that can obtain
Local derviation is sought again, can be obtained after abbreviation：
Wherein the first row is element entry identical with isomorphism data, is calculated in formula meter (10), the second row
It is exactly isomeric data, so information source pushes isomeric data formula after updating is：
Due in isomeric data, the number of the element of every kind of cloud data be not necessarily, so require to look up each N,
Due to the presence of transposed matrix, it is only necessary to transmit the data of half.Therefore transmission complexity is：
O (M, d)=(M-1) (d2+2d)
The isomeric data of this some of complex pushes, and MDCP model will call the free cloud of disclosed third party's open source on internet
Push Service (such as Baidu's cloud push) is pushed.The message push service of third-party platform is completely free, can be with zero one-tenth
This use, and possess powerful server cluster, there is high handling capacity, the message that user subscribes to can be at faster speed
It is sent to user terminal
3.4 MDCP model running processes
By above-mentioned analysis, original supplying system is difficult to realize the real-time high-efficiency push to multi-source heterogeneous data really,
After the obtaining needing to push of the task, MDCP model is called.
When message enters push list, MDCP model isolates multi-source isomorphism data by characteristic value calculating and multi-source is different
Structure data carry out the determination of weight to cloud data first, and the size of weight is allocated by system according to the significance level of attribute,
Attribute is separated according to weight again, if multi-source isomorphism data, then mode is pushed using third party, if multi-source heterogeneous number
According to then being pushed using MQTT, if can not push, reenters and wait channel.
According to the cyclicity in the period, MDCP model is used to filter out repeated data and delta data information, further according to
Feature vector distinguishes multi-source isomorphism data or multi-source heterogeneous data, the quantity for the system finally subscribed to according to user, network
Bandwidth, push mode used in the comprehensive descisions such as quantity and confidentiality of the data of push.
Further, the cloud push frame is divided into as follows towards the logical architecture of multi-source heterogeneous data cloud push platform
Five layers：
4.1 cloud data Layers：
This layer is the major part in cloud push platform, and the multi-source heterogeneous data in each system form cloud data, institute
There are data to belong to the layer, realize the logical Virtual of data, the multi-source heterogeneous data acquired in second step are to come to be somebody's turn to do
Layer.
4.2 data management layer：
This layer is the core of framework, realizes the collaborative work of data and push in cloud push, externally also providing has phase
With the access service of data.The layer is connected with cloud data Layer, and after the completion of data acquisition, the choosing of push is completed by MDCP model
It selects, while carrying out the grouping of data according to application-interface layer.
4.3 application-interface layer
The layer can configure on demand, according to different application or module that user subscribes to, provide what different data stored and accessed
Application service, such as wage Push Service, school grade service, information service of speculating in shares, video monitoring service etc., these service logical
Application-interface layer is crossed to interact with cloud data Layer.
4.4 access layer：
Authorized user can access cloud supplying system by the interface of standard, according to the different user class in client layer
Type, gives different user rights, and the mode of all types of user access is also not exactly the same.
4.5 client layer:
Client layer is mainly responsible for all kinds of registration users of management, while being also responsible for the modification of user information, updates, inquiry etc.,
All kinds of registration terminals are also managed simultaneously, such as：PC, IOS, Android etc..
The cloud method for pushing that the present invention designs is towards multi-source heterogeneous data, by the characteristic value and feature vector that calculate data
Come quick separating isomorphism data and isomeric data, to realize the efficient push of data.(MDCP is pushed by design Multidimensional decision-making cloud
Multiple decision cloud push) model, so that cloud push is not only reduced the synchronization in data-pushing renewal process
Data volume shortens the time, and solves the problems, such as the cross-platform push of multi-source heterogeneous data, huge particular for data volume
Greatly, and work working efficiency under low bandwidth, the insecure situation of network is obviously improved.
The present invention utilizes multi-source heterogeneous data cloud push technology, solves the deficiency in current supplying system.With existing skill
Art, which is compared, to be had the following advantages：
1. being suitable for the high push occasion of requirement of real-time
2. can analyze grouping test push function according to the label of user property, carry out intelligently pushing
3. the process of push is easy to operate, precise and high efficiency.
4. cloud push server real-time perception data are abnormal, when propelling data variation, server active push information evidence
To the end APP, the efficient of push is embodied, in real time.
Detailed description of the invention
Fig. 1 is the schematic diagram of cloud push platform feature design.
Fig. 2 is the schematic diagram of MDCP model running process.
Fig. 3 is the schematic diagram of cloud push Process Design.
Fig. 4 is the schematic diagram of cloud push logical architecture.
Fig. 5 is communication rate variation diagram.
Fig. 6 is cloud method for pushing flow diagram.
Fig. 7 is the schematic diagram of tradition push platform retention ratio.
Fig. 8 is the schematic diagram of cloud push platform retention ratio.
Specific embodiment
The present invention will be further described with reference to the accompanying drawing.
Embodiment 1
A kind of referring to Fig.1~Fig. 8, cloud method for pushing towards multi-source heterogeneous data, includes the following steps：
The first step：It designs cloud and pushes platform, functional module each in platform is parsed, cloud pushes platform feature design
As shown in Figure 1
1.1 clouds push platform can push the message of three types：Notice, transparent transmission message and Rich Media.Notice is to shifting
Moved end pushes the notification message for being presented in system notification bar.Transparent transmission message, which refers to, is sent customized content in a manner of transparent transmission
To client.Developer can set rule in client in advance, carry out message customization.Rich Media refer to push picture,
The information of the forms such as video, audio, network address.
1.2 clouds push platform should be supported to disappear to all users or according to labeling to the push of specific user group simultaneously
User grouping is given grouping user different labels by breath according to the message content that user subscribes to, and according to different labels, is carried out
The push of corresponding contents carries out single label to single user, is finally pushed according to label substance.
1.3 clouds push platform and provide user information and notification message statistical information, are counted and are believed according to the feedback information of user
Cease retention ratio, communication rate, flow consumption etc.
1.4 clouds push platform cross-platform can use (PC, iOS, Android), and user can also be according to their own needs
Add custom feature.
Second step：The collection process of multi-source heterogeneous data, as shown in Figure 3
Multi-source heterogeneous data in other each systems are considered as cloud data by 2.1, according to the information that user subscribes to, by cloud number
According to being classified as different modules.
The module information that 2.2 clouds push platform subscribes to user is managed, including subscribed content, registration information, modification
Information etc..
2.3 cloud push server are managed cloud data information, the update including cloud data, modification etc..
2.4 clouds push platform pre-processes the more new information in (2.3), the format including data, size etc..
Third step：Information collect after the completion of, need MDCP model to separate cloud data, be divided into multi-source heterogeneous data and
Multi-source isomorphism data, process are as shown in Figure 3：
3.1 multi-source heterogeneous data component analyses
Problem to be solved of the present invention is exactly the cloud data that are distributed on different Cloud Servers using these to be counted
According to principal component, pass through each key data signal component value XiSubtract each other the deviant in this direction as the ingredient with the μ provided,
By the main component value X of each datai(i=1,2...n) it inserts in matrix S.
S=[x1,x2,x3...xn] (1)
It sums after the diagonal matrix ST by matrix S multiplied by it and is averaged again again, since the Matrix Multiplication is with its diagonal matrix
It obtains being determining numerical value afterwards, so the value can be used to indicate whole isomery degree, be indicated with V.
Wherein μ is the weighted value obtained according to assessment, obtains its value range according to different systems, and N is sample total,
Since V is determining numerical value at this time, the isomery degree available feature value V of the sample is indicated.
It needs to push if there is multi-source heterogeneous data, the principal component of each data cannot be divided into X at this time1~XN, need at this time
Want XiBecome XIJ, so matrix S becomes：
At this point, V is matrix, it is considered as eigenmatrix, so diagonal matrix of the V multiplied by it, can release the feature of cloud data
Value k：
|V-kE| (4)
V gives feature vector, has found out corresponding characteristic value k, the size and V common metrics of characteristic value k by abbreviation
The isomery degree of different cloud data.
But the problem of data transfer load is not accounted in this method, different cloud data transmission in distributed environment
By communication rate, the limitation of the factors such as computing capability, it is desirable to reduce the data traffic between Cloud Server, avoid directly transmitting
Data can be carried out in a matrix fashion permutation and combination, then carry out cloud push based on this data structure by great amount of samples.
The push of 3.2 multi-source isomorphism data
In cloud data, although there is a large amount of multi-source heterogeneous data, there is also magnanimity multi-source isomorphism data,
When the push content that user needs belongs to isomorphism data, data are pressed into eigendecomposition using covariance matrix, are decomposed same
Structure data can use more efficient push mode, do not need to iterate in each point, saved data communication rates.
Firstly, with sample estimation local variance and mean vector μ in first cloud data station1, then obtain second
Sample estimation local variance and mean vector μ in cloud data station2, until obtaining the sample N of all cloud data stationsμ。Nkμk
For the remaining sample not updated, μKFor sample NKMean vector.In the prescribed limit of local variance and mean vector, then recognize
To be that these cloud data stations provide isomorphism data, average mean vector μ is found out, then pass through NμAnd NkμkFind out average ginseng
Number covariance：
Therefore in the case where each cloud data sample is sufficient, calculate isomorphism data accuracy be it is very big, due to
It is isomorphism data, secondary incremental update equation is:
Wherein (K=1,2,3.....).Due in above-mentioned algorithm, it can be noted that covariance matrix be it is symmetrical, because
During push, isomorphism data update every time for this, it is only necessary to transmit half of covariance matrix, MDCP model use is based on
The MQTT of WebSocket is pushed, if number of systems is M, the time complexity of number of parameters d, the algorithm are：
O (M, d)=(M-1) (d (d+1)/2+d+1)
The push of 3.3 multi-source heterogeneous data
Multi-source heterogeneous data are extended to isomorphism data main component, in order to facilitate mathematical computations, introduce displacement square
Battle array P, carries out displacement mapping to the sample obtained on Cloud Server, is as a result denoted as y, i.e.,：
Y=(ya,yb)t=PmX (9)
The purpose is to focus on before vector in current sample with identical part in isomorphism data, y is usedaIt indicates,
And different are partially disposed in behind vector, use ybIt indicates, equally, Mean Matrix and covariance matrix is replaced,
As a result it is denoted as μ, ∑ respectively：
μ=(μa,μb)t=Pmμ
μ and permutation matrix are all the last amounts being calculated, and Σ a is the covariance matrix obtained in m sample,
Σ c be from the covariance matrix of elements different in isomorphism data, and Σ b be between them covariance constitute matrix.
The formula meets higher-dimension distribution, so cloud data sample can be calculated using higher-dimension distribution, expression formula is such as
Under:
Y, μ, ∑ is brought by (5) (9) (10) 3 formula to be simplified, and takes natural logrithm that can obtain
Local derviation is sought again, can be obtained after abbreviation：
Wherein the first row is element entry identical with isomorphism data, is calculated in formula meter (10), the second row
It is exactly isomeric data, so information source pushes isomeric data formula after updating is：
Due in isomeric data, the number of the element of every kind of cloud data be not necessarily, so require to look up each N,
Due to the presence of transposed matrix, it is only necessary to transmit the data of half.Therefore transmission complexity is：
O (M, d)=(M-1) (d2+2d)
The isomeric data of this some of complex pushes, and MDCP model will call the free cloud of disclosed third party's open source on internet
Push Service (such as Baidu's cloud push) is pushed.The message push service of third-party platform is completely free, can be with zero one-tenth
This use, and possess powerful server cluster, there is high handling capacity, the message that user subscribes to can be at faster speed
It is sent to user terminal.
3.4MDCP model running process
By above-mentioned analysis, original supplying system is difficult to realize the real-time high-efficiency push to multi-source heterogeneous data really,
After the obtaining needing to push of the task, MDCP model is called, operational process is as shown in Figure 2.When message enters push list
When, MDCP model isolates multi-source isomorphism data and multi-source heterogeneous data by characteristic value calculating, if multi-source isomorphism data, then
Using MQTT protocol propelling (Fig. 2 is 1.), if multi-source heterogeneous data then use third party to push mode (Fig. 2 is 2.), if can not push away
It send, then reenters and wait channel (Fig. 2 is 3.).
4th step：Cloud push frame is the logical architecture figure towards multi-source heterogeneous data cloud push platform, as shown in figure 4,
Cloud push platform logic framework generally can be divided into following five layers：
4.1 cloud data Layers：
This layer is the major part in cloud push platform, and the multi-source heterogeneous data in each system form cloud data, institute
There are data to belong to the layer, realize the logical Virtual of data, the multi-source heterogeneous data acquired in second step are to come to be somebody's turn to do
Layer.
4.2 data management layer：
This layer is the core of framework, realizes the collaborative work of data and push in cloud push, externally also providing has phase
With the access service of data.The layer is connected with cloud data Layer, and after the completion of data acquisition, the choosing of push is completed by MDCP model
It selects, while carrying out the grouping of data according to application-interface layer.
4.3 application-interface layer
The layer can configure on demand, according to different application or module that user subscribes to, provide what different data stored and accessed
Application service, such as wage Push Service, school grade service, information service of speculating in shares, video monitoring service etc., these service logical
Application-interface layer is crossed to interact with cloud data Layer.
4.4 access layer：
Authorized user can access cloud supplying system by the interface of standard, according to the different user class in client layer
Type, gives different user rights, and the mode of all types of user access is also not exactly the same.
4.5 client layer:
Client layer is mainly responsible for all kinds of registration users of management, while being also responsible for the modification of user information, updates, inquiry etc.,
All kinds of registration terminals are also managed simultaneously, such as：PC, IOS, Android etc..
Embodiment 2
The present invention provides a kind of cloud method for pushing towards multi-source heterogeneous data, are applied to Android and iOS two
Large platform includes the following steps：
The first step：Cloud method for pushing is used on Android and iOS platform.
1.1 present invention are tested in Android and iOS using cloud push platform, cloud data source in result system,
Pay system and lithographic viewing system, collect after data and are by the main component value weight of data：User right, when user logs in
Between, the mode (one-to-many/one-to-one) of user's publish/subscribe, user logs in quantity, transmitted data amount.Weight mu is by synthesis
Consider and determine, it is assumed that in this system signal component value weight be (0.3,0.2,0.2,0.1,0.1,0.1).Weight is higher, this at
Influence of the variation of score value to last MDCP push result is bigger.Since data source is in three systems, and lithographic viewing system
Data type it is different (there are rich media informations), so collected data belong to multi-source heterogeneous data, as shown in table 1.
1 cloud Data Data signal component value of table
Data in table one are brought into specific embodiment 1 and can be obtained in formula (3) by 1.2：
Matrix S and weighted value μ is substituted into specific embodiment 1 can obtain in formula (2)：
After obtaining eigenmatrix, characteristic value can be found out by formula (4) in specific embodiment 1
It can be obtained after abbreviation：
K1=1.2 k2=1.6 k3=4.9
K3=4.9 obviously belongs to isomeric data, and k1, k2 can be considered as multi-source isomorphism data.
1.3
The data component value mean value of each period of table 2
As shown in Table 2, in order to acquire Mean Matrix μ, 6 class means is measured by 6 different time sections, constitute matrix μ：
MDCP finally determines comprehensive push mode, since weight is k=(0.3,0.2,0.2,0.1,0.1), so l (μ,
Σ y)=Σ | yk
μ, Σ y are substituted into 1 formula of specific embodiment (12), show that higher-dimension is distributed N=(L, μ, Σ), N is calculated according to vector
=(0.35,0.25,0.15).L is substituted into 1 formula of specific embodiment (13), it can be deduced that：
Since each humping section is the data updated, so 3 column redundant datas below in removal matrix, obtaining Σ d is
More new data：
As can be seen that Σ d is symmetrical matrix from matrix, so it is that original tradition pushes away that update, which is the data volume of transmission, every time
The half for sending mode, substantially increases pushing efficiency.
Second step：Cloud pushes platform and provides user information and notification message statistical information, is united according to the feedback information of user
Count information leave-on rate, communication rate, flow consumption etc.
The analysis of 2.1 communication rates, as shown in figure 5, abscissa indicates number of tasks, ordinate indicates communication rate
For the ability for measuring MDCP algorithm proposed by the present invention, from average transmission rate, communication rate, flow when standing, push
Retention ratio assesses the model validation in multi-source heterogeneous data cloud method for pushing.
Communication rate refers to the number of communications of user and cloud push platform in the unit time, and whether test user is ready to use this
Platform carries out data-pushing, and tests can or can not generate other problems when number of communications is high simultaneously, traditional push
Mode communication rate, which is basically unchanged, is asked due to can all send to server always during the entire process of pushing task queue message
It asks, and cloud proposed by the present invention push model is in information collecting step, with the operation of system, task quantity increases, advantage
It just gradually shows, more in number of tasks, the traffic of cost tails off instead, as shown in table 3.
3 average transmission rate tables of data of table
2.2 cloud method for pushing flows, as shown in fig. 6, abscissa indicates that flow value, ordinate indicate time of repose.
Flow refers to the added flow that generates due to push when mobile phone is stood when standing, and whether cloud pushes platform can be because
For communication rate improvement and generate a large amount of flows, be compared respectively using cloud push platform and tradition push platform, as a result table
Bright cloud push platform is in the longer situation down-off consumption of mobile device time of repose less than tradition push platform.Average transmission rate is
Refer to that the transmission quantity to compare with common push mode, the results are shown in Table 4 by multiple comparing calculation：
Current capacity contrast's table when table 3 is stood
2.3 cloud method for pushing retention ratios, as shown in Figure 7 and Figure 8, abscissa indicate number of weeks, and ordinate indicates retention ratio
Percentage.
Retention ratio refers to that within a certain period of time, such as 1-6 week, also there are the ratios of the PUSH message by user, also can be from
Reflect influence of the model to user to a certain degree, tradition push platform and cloud push platform retention ratio comparison diagram, as a result table
The retention ratio of bright cloud push platform almost per week is both greater than one times or more of tradition push platform retention ratio, passes after 6 week
The retention ratio of system push platform is less than 20%, and cloud push platform nearly reaches 50%.
Claims (2)
1. a kind of cloud method for pushing towards multi-source heterogeneous data, it is characterised in that：The cloud method for pushing includes the following steps：
The first step：It designs cloud and pushes platform, process is as follows：
1.1 clouds push the message of platform push three types：Notice, transparent transmission message and Rich Media；
1.2 clouds push platform should be supported simultaneously to all users or according to labeling to specific user group PUSH message, root
It gives grouping user different labels user grouping according to the message content that user subscribes to, according to different labels, carries out corresponding
The push of content carries out single label to single user, is finally pushed according to label substance；
1.3 clouds push platform and provide user information and notification message statistical information, are stayed according to the feedback information statistical information of user
Deposit rate, communication rate and flow consumption；
1.4 clouds push the cross-platform use of platform, and user adds custom feature according to their own needs；
Second step：The acquisition of multi-source heterogeneous data, process are as follows：
Multi-source heterogeneous data in environment not in homologous ray are considered as cloud data by 2.1, the different weights given according to system, by cloud
Data classification is different modules, including notification module, Rich Media's module and transparent transmission message module；
2.2 clouds push platform is managed according to the module information that user subscribes to, according to the module that different user is subscribed to, by user
It is grouped, the user with equal modules is while to manage the information of registration user with group and labelled；
2.3 cloud push server are managed cloud data information, and information is managed according to the time, records each update
Data, use when to push in next step；
2.4 clouds push platform pre-processes the more new information in (2.3), and data format is identical, and weight is identical to be divided into
Different grouping facilitates the separation of isomorphism data and isomeric data, and MDCP model is by the data of access to information database, by determining weight
It is separated with attribute and carries out decision push, as entire " subscription-collection-decision-push " period；
Third step：After the completion of information is collected, needs MDCP model to separate cloud data, be divided into multi-source heterogeneous data and multi-source
Isomorphism data, process are as follows：
3.1 multi-source heterogeneous data component analyses
The isomeric data on different Cloud Servers is distributed in obtain the principal component of data using these, passes through each key data
Signal component value XiSubtract each other the deviant as the ingredient in the vector direction of μ with the μ provided, by each key data signal component value Xi
It inserts in matrix S, i=1,2...N；
S=[X1,X2,X3,...,XN] (1)
It sums after the diagonal matrix ST by matrix S multiplied by it and is averaged again again, since the Matrix Multiplication after its diagonal matrix to obtain
Arriving is determining numerical value, so the value is used to indicate whole isomery degree, is indicated with V：
Wherein μ is the weighted value obtained according to assessment, obtains its value range according to different systems, and N is sample total, due to
V is determining numerical value at this time, and the isomery degree available feature value V of the sample is indicated；
It needs to push if there is multi-source heterogeneous data, the principal component of each data cannot be divided into X at this time1~XN, X is needed at this timei
Become Xij, so matrix S becomes：
At this point, V is matrix, it is considered as eigenmatrix, so diagonal matrix of the V multiplied by it, can release the characteristic value k of cloud data：
|V-kE| (4)
V gives feature vector, has found out corresponding characteristic value k by abbreviation, and the size of characteristic value k and V common metrics are not
With the isomery degree of cloud data；
The push of 3.2 multi-source isomorphism data, process are as follows：
Firstly, with sample estimation local variance and mean vector μ in first cloud data station1, then obtain second cloud data
Sample estimation local variance and mean vector μ on website2, until obtaining the sample N of all cloud data stationsK, in local variance
In the prescribed limit of mean vector, then it is assumed that be that these cloud data stations provide isomorphism data, find out average mean to
AmountPass through againFind out mean parameter covariance：
Due to being isomorphism data, secondary incremental update equation is:
Wherein, K=1,2,3....., during push, isomorphism data update every time, it is only necessary to transmit half of covariance square
Battle array, MDCP model are pushed using the MQTT based on WebSocket, if number of systems is M, number of parameters d, time complexity
Degree is：
O (M, d)=(M-1) (d (d+1)/2+d+1)；
The push of 3.3 multi-source heterogeneous data
Multi-source heterogeneous data are extended to isomorphism data main component, permutation matrix P are introduced, to what is obtained on Cloud Server
Sample carries out displacement mapping, is as a result denoted as y, i.e.,：
Y=(ya,yb)t=PmX (9)
It will be focused on before vector in current sample with identical part in isomorphism data, use yaIt indicates, and by different parts
It is placed on behind vector, uses ybIt indicates, equally, Mean Matrix and covariance matrix is replaced, are as a result denoted as respectively
∑：
μ and permutation matrix are all the last amounts being calculated, and Σ a is the covariance matrix obtained in m sample, Σ c
Be from the covariance matrix of elements different in isomorphism data, and Σ b be between them covariance constitute matrix；
Cloud data sample is calculated using higher-dimension distribution, and expression formula is as follows:
Y, μ, ∑ is brought by (5) (9) (10) 3 formula to be simplified, and natural logrithm is taken to obtain
Local derviation is sought again, after abbreviation：
Wherein the first row is element entry identical with isomorphism data, is calculated in formula (10), the second row is exactly different
Structure data, so information source pushes isomeric data formula after updating is：
3.4 MDCP model running process,
After the obtaining needing to push of the task, MDCP model is called,
When message enters push list, MDCP model isolates multi-source isomorphism data and multi-source heterogeneous number by characteristic value calculating
According to, first cloud data are carried out with the determination of weight, the size of weight is allocated by system according to the significance level of attribute, then by
Attribute is separated according to weight, if multi-source isomorphism data, then mode is pushed using third party, if multi-source heterogeneous data are then
It is pushed using MQTT, if can not push, reenters and wait channel；
According to the cyclicity in period, MDCP model is used to filter out repeated data and delta data information, further according to feature to
Amount distinguishes multi-source isomorphism data or multi-source heterogeneous data, the quantity for the system finally subscribed to according to user, the bandwidth of network,
Push mode used in the quantity and confidentiality comprehensive descision of the data of push.
2. a kind of cloud method for pushing towards multi-source heterogeneous data as described in claim 1, it is characterised in that：The cloud push
Frame is to be divided into following five layers towards the logical architecture of multi-source heterogeneous data cloud push platform：
4.1 cloud data Layers：Multi-source heterogeneous data in each system form cloud data, and all data belong to the layer, realize
The logical Virtual of data, the multi-source heterogeneous data acquired in second step are from the layer；
4.2 data management layer：The collaborative work for realizing data and push in cloud push, externally also provides the visit with identical data
The service of asking, the layer are connected with cloud data Layer, and after the completion of data acquisition, the selection of push, while root are completed by MDCP model
The grouping of data is carried out according to application-interface layer；
4.3 application-interface layer：The layer can configure on demand, according to different application or module that user subscribes to, provide different data and deposit
The application service of storage and access, these services are interacted by application-interface layer with cloud data Layer；
4.4 access layer：Authorized user can access cloud supplying system by the interface of standard, be used according to the difference in client layer
Family type, gives different user rights, and the mode of all types of user access is also not exactly the same；
4.5 client layer：It is responsible for all kinds of registration users of management, while is also responsible for the modification of user information, updates and inquiring, while
Manage all kinds of registration terminals.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610077551.7A CN105760449B (en) | 2016-02-03 | 2016-02-03 | A kind of cloud method for pushing towards multi-source heterogeneous data |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610077551.7A CN105760449B (en) | 2016-02-03 | 2016-02-03 | A kind of cloud method for pushing towards multi-source heterogeneous data |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN105760449A CN105760449A (en) | 2016-07-13 |
| CN105760449B true CN105760449B (en) | 2018-11-30 |
Family
ID=56329938
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610077551.7A Active CN105760449B (en) | 2016-02-03 | 2016-02-03 | A kind of cloud method for pushing towards multi-source heterogeneous data |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN105760449B (en) |
Families Citing this family (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106528812B (en) * | 2016-08-05 | 2019-04-23 | 浙江工业大学 | A kind of cloud recommended method based on USDR model |
| CN106412101A (en) * | 2016-10-28 | 2017-02-15 | 浪潮软件集团有限公司 | MQTT protocol-based user-oriented message pushing method |
| CN106775938A (en) * | 2016-12-04 | 2017-05-31 | 国云科技股份有限公司 | A kind of virtualization data delivery system and its implementation |
| CN107341222B (en) * | 2017-06-28 | 2020-04-07 | 清华大学 | Cross-platform theme association method, device and equipment |
| CN107741875B (en) * | 2017-10-20 | 2020-08-21 | 北京易思捷信息技术有限公司 | Heterogeneous management system |
| CN108055306A (en) * | 2017-12-06 | 2018-05-18 | 深圳市智物联网络有限公司 | Data processing method, relevant device and system in a kind of Internet of Things |
| CN108287889B (en) * | 2018-01-17 | 2019-06-18 | 清华大学 | A kind of multi-source heterogeneous date storage method and system based on elastic table model |
| CN108959603B (en) * | 2018-07-13 | 2022-03-29 | 北京印刷学院 | Personalized recommendation system and method based on deep neural network |
| CN114513335B (en) * | 2022-01-18 | 2022-11-29 | 郑州大学 | Data flow fusion efficient transmission method based on one-way optical gate |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104809553A (en) * | 2015-04-20 | 2015-07-29 | 广东工业大学 | Multi-source electronic commerce data processing platform and method for heterogeneous data |
| CN104967631A (en) * | 2014-04-17 | 2015-10-07 | 腾讯科技（深圳）有限公司 | Method for sharing cloud storage content, and apparatus thereof |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9165034B2 (en) * | 2009-10-15 | 2015-10-20 | Hewlett-Packard Development Company, L.P. | Heterogeneous data source management |
-
2016
- 2016-02-03 CN CN201610077551.7A patent/CN105760449B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104967631A (en) * | 2014-04-17 | 2015-10-07 | 腾讯科技（深圳）有限公司 | Method for sharing cloud storage content, and apparatus thereof |
| CN104809553A (en) * | 2015-04-20 | 2015-07-29 | 广东工业大学 | Multi-source electronic commerce data processing platform and method for heterogeneous data |
Non-Patent Citations (2)
| Title |
|---|
| Practical Resource Provisioning and Caching with Dynamic Resilience for Cloud-Based Content Distribution Networks;Menglan Hu et al;《IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS》;20140831;第25卷(第8期);第2169-2179页 * |
| 基于模糊综合决策的服务器推送方法;蒋乾悦 等;《计算机科学》;20140531;第41卷(第5期);第86-90页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN105760449A (en) | 2016-07-13 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN105760449B (en) | A kind of cloud method for pushing towards multi-source heterogeneous data | |
| US10652633B2 (en) | Integrated solutions of Internet of Things and smart grid network pertaining to communication, data and asset serialization, and data modeling algorithms | |
| CN102143507B (en) | Method and system for monitoring service quality, and analytical method and system therefor | |
| US8156172B2 (en) | Monitoring and reporting enterprise data using a message-based data exchange | |
| US20120130940A1 (en) | Real-time analytics of streaming data | |
| US10528120B2 (en) | Customized internet-of-things data packaging and brokering | |
| US20160140580A1 (en) | Customer demographic information system and method | |
| WO2011103060A2 (en) | Service provider recommendation engine | |
| CN109254901A (en) | A kind of Monitoring Indexes method and system | |
| CN102056351A (en) | Push service system and method | |
| CN112464123A (en) | Water quality monitoring data visualization system and method based on micro-service | |
| Lohitha et al. | Integrated publish/subscribe and push-pull method for cloud based IoT framework for real time data processing | |
| CN107249034A (en) | A kind of self-service platform and method based on cloud computing | |
| CN106817710A (en) | The localization method and device of a kind of network problem | |
| CN116089490A (en) | Data analysis method, device, terminal and storage medium | |
| CN116319404A (en) | Multifunctional monitoring system management system for mobile phone application | |
| CN115086180B (en) | Network networking method, network networking device and electronic equipment | |
| CN116910144A (en) | Computing power network resource center, computing power service system and data processing method | |
| CN103188629B (en) | Flow bootstrap technique between a kind of networks with different systems and device | |
| CN104079627B (en) | Send the method and apparatus for showing information | |
| CN114185698A (en) | Remote sensing data processing system based on micro-service and RPC communication | |
| Du et al. | A QoE based evaluation of service quality in wireless communication network | |
| CN101605339A (en) | Monitoring of network bandwidth resources operating position and prompt system and method | |
| CN110245138A (en) | A kind of area data management system | |
| Chen | Neighborhood optimization of intelligent wireless mobile network based on big data technology |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |