CN104484673A - Data complementation method for pattern recognition application of real-time data flow - Google Patents

Abstract

The invention discloses a data complementation method of pattern recognition application of real-time data flow. The data complementation method comprises a runtime dynamic cache, a data integrity index and a sliding-window-based iterative data complementation algorithm, wherein the runtime dynamic cache is used for saving obtained real-time data; the data integrity index is used for judging the data integrity according to the data rate and the distribution of data in a current sliding window; the data complementation algorithm is used for iteratively using the cached history data to complement the data of the current sliding window according to the data integrity. The invention discloses the high-efficiency data complementation method for the common data loss problem in the pattern recognition application of the real-time data flow; the method has the advantages of being good in complementation effect, high in online degree, wide in application range and transparent to application; the method can be effectively applied to the various pattern recognition application for continuous real-time data flow.

Description

The Supplementing Data method of real-time stream application of pattern recognition

Technical field

The present invention relates to a kind of Supplementing Data method for real-time stream application of pattern recognition, belong to computer application field, for the shortage of data problem caused because of reasons such as equipment performance restriction, network communication quality fluctuations common in real-time stream sample and transform, in conjunction with the characteristic of online real-time mode recognizing application, achieve a kind of Supplementing Data method efficiently.

Background technology

Along with Internet of Things correlation technique, as the fast development of wireless sensor network, wearable perception and computing equipment, computer system constantly can obtain external environment condition perception data and to go forward side by side row relax.These data uninterruptedly perceived by a large amount of sensor node to processing node, just define potential infinite real-time stream by transmission means Systems such as wireless networks.These real-time streams after treatment, by technology such as pattern-recognitions, therefrom can extract and have high abstraction hierarchy, abundant semantic, useful information, serve the mankind.

Due to the unreliability of wireless network transmissions, the reason such as limitation, various kinds of sensors timed sleep of bandwidth, often there is the disappearance of data in the real-time stream that processing node receives.And apply different from Real-time digital signal processing and streaming media playing class, algorithm for pattern recognition is intended to extract useful feature from bottom data stream, thus identifies and have high abstraction hierarchy, abundant semantic, useful information.Supplementing Data method is the important technology of tackling shortage of data problem in application of pattern recognition, and traditional method often adopts the mode inserting the data of fixed default value or recent acquisition at disappearance place to carry out completion to missing data.But because this method does not agree with the application of pattern-recognition class well, therefore its effect is not often remarkable especially.

Summary of the invention

Goal of the invention: in order to overcome the deficiency of classic method Supplementing Data weak effect, the feature that the present invention applies from the pattern-recognition class for real-time streaming data, propose a kind of use cache line data, low redundancy, efficiently Supplementing Data method.Efficient while, this Supplementing Data method does not do any presetting to input data characteristics, follow-up mode recognition methods and other system component implementation, can insert seamless, pellucidly and extract application system flow chart of data processing, meet the demand of different application scene flexibly.

Say from the general extent, the application of pattern-recognition class has following three features: 1) responsive to shortage of data: the information dropout that shortage of data causes, and can cause have a strong impact on the quality of feature extraction, and then the accuracy of Effect Mode recognizer; 2) from pattern recognition result angle, data have localisation features: higher level of abstraction Informational Expression is comprise the classification enriching semantic information, different from real-time raw data, the change frequency of classification is relatively slow, therefore, when the classification that consideration data are corresponding, present the characteristic (that is, the classification corresponding to data does not change within a period of time) of localization; 3) redundant information is worth little: repetition, the data of redundancy are little for the value extracting useful feature.

According to above three features, we determine the Supplementing Data algorithm requirements for application of pattern recognition, propose the method utilizing historical data to carry out completion, propose the Supplementing Data algorithm of corresponding integrity metrics and anti-redundancy.Specifically, the Supplementing Data method for real-time streaming data application of pattern recognition proposed by the invention, comprise one run time data cache line, an item number is according to integrity metrics and an iterative anti-redundant data completion algorithm based on moving window.This Supplementing Data method is transparent for other assemblies of application of pattern recognition, can be conveniently used in the anti-loss of data ability strengthening system in the system of having disposed, and also can close at any time when needed and the normal operation of not other assemblies of influential system.

Technical scheme: a kind of Supplementing Data method of real-time stream application of pattern recognition, its detailed process comprises following steps:

1) data source produces data continuously and forms input traffic, if data transfer rate is known as a r per second sampled value (r determines according to application scenarios, is arithmetic number), data cache line when input data are stored in operation, remembers that the data of buffer memory are D;

2) with current time t for starting point, use size in the data of buffer memory, to intercept forward one section of D [t-w, t] as current window data for the moving window of w second (w determines according to application scenarios, is arithmetic number);

3) on the basis of current window data D [t-w, t], in conjunction with data transfer rate r, the moving window size w in given data source, data integrity index c is calculated;

4) if current window data D is [t-w, t] integrity metrics c not up to standard, then from the data D [t-2w of previous window, t-w] in be principle with minimal redundancy, choose data filling and insert current window D [t-w, t], if integrity metrics c reaches standard after completion, algorithm terminates, otherwise iteration performs this step until integrity metrics c reaches all data in standard or limit buffer memory.

Wherein, step 1) detailed process be:

1.1) buffer memory D when creating the on-line operation of dynamic size;

1.2) data data source produced stored in D, and according to data obtaining time sort ascending;

1.3) according to the demand of upper mode identification application, in definition D, the longest data cached retention time is h second (h is the integral multiple of w), to avoid the data that buffer memory is too outmoded.

Wherein, step 2) detailed process be:

2.1) according to the demand of upper mode identification application, size w second of definition moving window is rational data burst granularities in time;

2.2) with current time t for starting point, forward trace reads in buffer memory D and starts the one piece of data D [t-w, t] that terminates to the t data as current sliding window mouth with the t-w moment.

Wherein, step 3) detailed process be:

3.1) given current sliding window mouth data D [t-w, t], the data transfer rate r in given data source and moving window size w, according to following formulae discovery data integrity index c:

Wherein, molecule | D [t-w, t] | to represent in D [t-w, t] comprise the quantity of data.According to above-mentioned formula, because denominator rw is the upper limit of w time inner sensor data volume, therefore c is for being defined in the real number on [0,1] interval, and c value is larger, represents that the integrality of data D [t-w, t] is higher.

Wherein, step 4) detailed process be:

4.1) completion traceback depth variable j is set and is initialized as j=1;

4.2) set buffer memory historical data section for completion as D [t-2w, t-w], get D [t-2w, t-w] in j data of afterbody and D [t-w, t] in j data of head compare, the absolute value phase adduction getting its difference is averaging, namely, according to the mean value of difference between following formulae discovery j item number certificate, d (j):

$d\left(j\right)=\frac{1}{j}\underset{k=1}{\overset{j}{\mathrm{\Σ}}}\left|D\right[t-w-j+k-1]-D[t-w+k-1\left]\right|$

Wherein, j is the lap size of historical data section and current data section, k gets 1 to j and represents and enumerate the data of lap, | D [t-w-j+k-1] – D [t-w+k-1] | represent that the Section 1 data of the jth item number certificate reciprocal of historical data section and current data section are to thereafter, the absolute value of the numerical difference of lap kth item number certificate;

4.3) calculate the value of the d (j) when the value of j is 1 to w, finally determine the value j ' of the j making d (j) minimum;

4.4) by the data subsequence in D [t-2w, t-w-j '], add to the front of data sequence in D [t-w, t] by former order, obtain the data D [t-w, t] in the current sliding window mouth after completion.

Accompanying drawing explanation

Fig. 1 is system flowchart;

Fig. 2 is Supplementing Data method flow diagram.

Embodiment

Below in conjunction with specific embodiment, illustrate the present invention further, these embodiments should be understood only be not used in for illustration of the present invention and limit the scope of the invention, after having read the present invention, the amendment of those skilled in the art to the various equivalent form of value of the present invention has all fallen within the application's claims limited range.

1, hardware environment

1) data source be made up of one or more sensor node, can produce sensing data continuously and pool data stream, the data in data stream may produce disappearance;

2) a Supplementing Data server, this server can be connected into data source and obtain real-time stream, and possesses the demand that enough Storage and Processing abilities (data transfer rate depending on data stream) meet data cache line and completion algorithm.

2, application scenarios

When applying Supplementing Data method disclosed in this invention, first need the sensor data stream access data completion server obtained by Real-time Collection.By burst size w, the data integrity index c and cache size h of user according to the demand specified data of follow-up mode identification method.Server, according to the real time data received, judges the integrality of data and carries out corresponding completion operation in units of the w time.This Supplementing Data method guarantees that the data stream exported reaches the data integrity index c of user's defined as much as possible, ensures the data of data for collecting in the up-to-date h time comprised simultaneously.Data after completion compare raw data higher integrality, is more conducive to follow-up mode identification method and therefrom extracts useful information.

The form that data after completion are identical with input traffic, therefore this Supplementing Data method to the operation of data stream for application other assemblies be transparent, can insert and extract the flow chart of data processing of application system easily, meet the demand of various flows Recognition system application.

A typical apply scene of technology involved in the present invention take technology of Internet of things as the large market demand supported.Under the support of Internet of Things infrastructure, can the data such as a large amount of environment, user health information be collected and be aggregated into high in the clouds.This kind of technology comprises the Detection of Air Quality network, vehicle-mounted mobile sensor network, intelligent wearable device network etc. that have been at present in the middle of Fast Construction.For intelligent wearable device network, emerging in the market take Intelligent bracelet as the wearable device of representative, with higher sample frequency, can obtain the data relevant with user's physiological health comprising acceleration, heart rate, body temperature.These raw data collected, by wireless communication technology, are aggregated into high in the clouds with the form of real-time stream and process.High in the clouds software, by a series of mode identification method, can identify the high-level information that the action, emotional state, daily life rule etc. of user are useful from user's physiology raw data, and then better for user provides service.But cause uncontrollable reasons such as shutdown, equipment active sleep, the improper use of user due to the unreliability of network service, running down of battery, bottom hardware gather and be sent in the raw data in high in the clouds and inevitably there is disappearance

Situation.To ensure high-quality, the high integrality of data from data source header, then to pay sizable cost or cannot realize at all.And pass through reasonable employment Supplementing Data method proposed by the invention, when raw data exists disappearance, with very little cost, the high-quality of high-rise pattern recognition result can be kept, meet the integrated demand of application.

3, method describes

Supplementing Data method for real-time stream application of pattern recognition involved in the present invention, its flow chart of data processing as shown in Figure 1.

1) data fragmentation and caching technology

Real time data complementing method involved in the present invention, its basis is burst to input traffic and caching technology.Suppose that current time is t, according to application demand, can determine that the size of time slicing is w second, the data carry mechanism that application of pattern recognition can be tolerated is h second (h is the integral multiple of w).First data fragmentation and caching technology utilize a length input traffic to be cut into time upper disjoint data segment D [t-w for w moving window second, t], D [t-2w, t-w] ..., D [i, j] ..., D [t-h, t-h+w], wherein i and j be respectively in data segment comprise start and end time of data.Due to the data volume comprised in every segment data, according to data transfer rate and the shortage of data situation of data source, may larger fluctuation be there is, therefore data cached section time, need to open up dynamic memory space and store data cached.Consider that the value of h is often less, therefore can complete in internal memory the buffer memory of data.

2) data integrity index calculate

The data transfer rate in tentation data source is a r per second sampled value, given current time t and burst size w, as the current data section D [t-w that burst obtains, t] in the item number of sampled data that comprises | D [t-w, t] | when being less than rw, claim the data in current data section imperfect, and the degree of its imperfect (or complete), portrayed by integrality quantizating index c, the computing method of c are shown below:

3) minimal redundancy Data Matching and complementing method

When the data volume comprised in current data section does not reach the data volume required for application system, namely, data integrity index c lower than application defined threshold value time (value of threshold value is by applying decision, it is more than 0.7 comparatively reasonable often to get), then need to carry out completion to the data in current data section.When carrying out completion, utilize the localisation features of data, namely, classification corresponding to data does not change within a period of time, thus the data filling that can pass through to collect in nearlyer a period of time is in existing data segment, realize introducing on the basis of other categorical datas with lower probability, supply the object of the information lacked in current data section.Meanwhile, according to the feature of application of pattern recognition, the value of redundant data is very low, therefore needs when completion to remove redundant information.

When completion, with the data overlap of a unit interval for starting point, the similarity degree of the historical data section calculating current data section and buffer memory when in various degree overlapping judges redundancy.Computation process for the similarity quantizating index judging degree of redundancy is as described below:

If be D [t-2w for the buffer memory historical data section of completion, t-w], get D [t-2w, t-w] in j data of afterbody and D [t-w, t] in j data of head compare, the absolute value phase adduction getting its difference is averaging, namely, according to the mean value of difference between following formulae discovery j item number certificate, d (j):

$d\left(j\right)=\frac{1}{j}\underset{k=1}{\overset{j}{\mathrm{\Σ}}}\left|D\right[t-w-j+k-1]-D[t-w+k-1\left]\right|$

Get the value of value j ' for final j of the j making d (j) minimum, and by the data subsequence in D [t-2w, t-w-j '], D [t-w is added to by former order, t] in the front of data sequence, obtain the data D [t-w, t] in the current sliding window mouth after completion.

After completing a current completion, calculate the data integrity index of D [t-w, t] once again, if do not reach the integrity thresholds of application defined, then continue to use this process, use the historical data of buffer memory in D [t-3w, t-2w] to carry out completion to the data in current data section; If data integrity reaches standard, or the data of all buffer memorys of limit, then terminate algorithm.

Claims (5)

1. the Supplementing Data method of a real-time stream application of pattern recognition, it is characterized in that, comprise data cache line when running, an item number based on the iterative anti-redundant data completion algorithm of moving window, specifically comprises following steps according to integrity metrics and one:

1) data source produces data continuously and forms input traffic, if data transfer rate is known as a r per second sampled value, data cache line when input data are stored in operation, remembers that the data of buffer memory are D;

2) with current time t for starting point, use the moving window that size is w second in the data of buffer memory, to intercept forward one section of D [t-w, t] as current window data;

3) on the basis of current window data D [t-w, t], in conjunction with data transfer rate r, the moving window size w in given data source, data integrity index c is calculated;

4) if current window data D is [t-w, t] integrity metrics c not up to standard, then from the data D [t-2w of previous window, t-w] in be principle with minimal redundancy, choose data filling and insert current window D [t-w, t], if integrity metrics c reaches standard after completion, algorithm terminates, otherwise iteration performs this step until integrity metrics c reaches all data in standard or limit buffer memory.

2. the Supplementing Data method of real-time stream application of pattern recognition as claimed in claim 1, is characterized in that, step 1) detailed process be:

1.1) buffer memory D when creating the on-line operation of dynamic size;

1.2) data data source produced stored in D, and according to data obtaining time sort ascending;

1.3) according to the demand of upper mode identification application, in definition D, the longest data cached retention time is h second, to avoid the data that buffer memory is too outmoded.

3. the Supplementing Data method of real-time stream application of pattern recognition as claimed in claim 2, is characterized in that, step 2) detailed process be:

2.1) according to the demand of upper mode identification application, size w second of definition moving window is rational data burst granularities in time;

2.2) with current time t for starting point, forward trace reads in buffer memory D and starts the one piece of data D [t-w, t] that terminates to the t data as current sliding window mouth with the t-w moment.

4. the Supplementing Data method of real-time stream application of pattern recognition as claimed in claim 2, is characterized in that, step 3) detailed process be:

3.1) given current sliding window mouth data D [t-w, t], the data transfer rate r in given data source and moving window size w, according to following formulae discovery data integrity index c:

Wherein, molecule | D [t-w, t] | to represent in D [t-w, t] comprise the quantity of data; According to above-mentioned formula, because denominator rw is the upper limit of w time inner sensor data volume, therefore c is for being defined in the real number on [0,1] interval, and c value is larger, represents that the integrality of data D [t-w, t] is higher.

5. the Supplementing Data method of real-time stream application of pattern recognition as claimed in claim 2, is characterized in that, step 4) detailed process be:

4.1) completion traceback depth variable j is set and is initialized as j=1;

4.2) set buffer memory historical data section for completion as D [t-2w, t-w], get D [t-2w, t-w] in j data of afterbody and D [t-w, t] in j data of head compare, the absolute value phase adduction getting its difference is averaging, namely, according to the mean value of difference between following formulae discovery j item number certificate, d (j):

$d\left(j\right)=\frac{1}{j}\underset{k=1}{\overset{j}{\mathrm{\Σ}}}\left|D\right[t-\mathrm{\ω}-j+k-1]-D[t-\mathrm{\ω}+k-1\left]\right|$

Wherein, j is the lap size of historical data section and current data section, k gets 1 to j and represents and enumerate the data of lap, | D [t-w-j+k-1] – D [t-w+k-1] | represent that the Section 1 data of the jth item number certificate reciprocal of historical data section and current data section are to thereafter, the absolute value of the numerical difference of lap kth item number certificate;

4.3) calculate the value of the d (j) when the value of j is 1 to w, finally determine the value j ' of the j making d (j) minimum;

4.4) by the data subsequence in D [t-2w, t-w-j '], add to the front of data sequence in D [t-w, t] by former order, obtain the data D [t-w, t] in the current sliding window mouth after completion.