CN106897705A - A kind of oceanographic observation big data location mode based on incremental learning - Google Patents

Abstract

The present invention relates to a kind of oceanographic observation big data location mode based on incremental learning, a kind of oceanographic observation big data location mode based on incremental learning, the location mode is comprised the following steps：S1：It is input into increment oceanographic observation data set to be laid out；S2：Initialization memory capacity；S3：Calculate the data value that incremental data concentrates data；S4：All data that incremental data is concentrated are divided；S5：Incremental data set is trained using Increment Learning Algorithm；S6：Data after training are laid out；S7：Increment oceanographic observation data set after output layout；Wherein, the Increment Learning Algorithm in described step S5 is SVMs Increment Learning Algorithm.The advantage is that, it is ensured that while classification accuracy rate, reduce the expense of training time and the response time of user accesses data；Solved the problems, such as using the Incremental Learning Algorithm of SVMs flux matched；It has been effectively compressed the size of sample set and has given up useless sample.

Description

A kind of oceanographic observation big data location mode based on incremental learning

Technical field

It is a kind of oceanographic observation based on incremental learning specifically the present invention relates to oceanographic data distribution technique field Big data location mode.

Background technology

With the iterative method of ocean power of China strategy, the fast development of science big data technology is marine economic industry It is filled with science power.Additionally, the upper rail of " ocean one " A stars and " ocean one " B magnitudes special topic satellite successfully optimizes me State's ocean three-dimensional observation road network so that high accuracy, frequent, the real-time multimode state oceanographic data of big covering are in quick-fried geometric progression Fried formula increases.The polyphyly of Marine Sciences Disciplinary features and oceanographic data obtaining means result in oceanographic data has magnanimity Property, multidimensional, in real time, the feature such as strong association so that oceanographic data turns into the model of big data.To oceanographic observation big data effectively It is the critical path for excavating oceanographic data value to be stored, managed and built ocean big data service.

Data distribution is the key issue in data storage, and it is that data are divided into a series of disjoint data slots Or region, and be placed on each back end according to certain data distribution strategy dispersion.During data distribution, well Burst implementation strategy be data distribution key.Existing data fragmentation strategy (as rotation is divided) is applied to stent The universal relation type database of formula, the effect is significant in conventional data.However, multi-modal Real-Time Ocean observation big data has Special property so that the characteristics of traditional stripping strategy have ignored itself when burst is carried out to oceanographic observation big data, Lack certain practicality.Therefore need to further consider and analyze the data value of oceanographic data itself, just can effectively to data It is distributed and is stored.

Additionally, developing rapidly with extensive ocean stereopsis technology, during actual oceanographic observation, Oceanic View The information for surveying big data is not disposable acquisition, can constantly have new data to increase.In face of significantly ever-increasing ocean Observation big data, if each time will the modeling storage or when carrying out data mining and need to spend substantial amounts of again in all data Between, this is clearly unpractical.And incremental learning can efficiently solve above mentioned problem so that the storage and management of ocean big data Serviceization, practical can preferably be moved towards.

The main purpose of data distribution is, by the reasonable layout of data, to deposit data in situ as much as possible, is reduced Across logical partition or the data access of physical node.

Under the Strategic Demand and novel information technology fast development of ocean power, ocean big data is excavated and managed Reason can provide important information for the research such as early-warning and predicting of the observation of marine environment, the detection of marine resources and Oceanic disasters Resource.Variation and deep layout however as oceanographic observation means and equipment, such as buoy, satellite, remote sensing, observation station are in real time Data source gather, cause data volume level of confidentiality increase so that traditional data distribution strategy for oceanographic data storage and Management produces certain limitation.

In face of the magnanimity oceanographic observation data of rapid growth, the result of historical data study how is effectively utilized, to new Increase data efficiently to be analyzed, so as to avoid repetition training and study to historical sample, obtain accurate data point Class result is the key being distributed to oceanographic observation data, and incremental learning can be good at solving problems.At present, increase Amount learning algorithm has obtained preferable application in some fields.In distributed process is carried out to oceanographic data, in face of in real time more New observation data, good dynamic adaptivity can bring to the response time of the distributed effect of data and user accesses data Preferably influence.Therefore, in face of the oceanographic observation big data of continuous real-time update, the thought of incremental learning is introduced into the big number in ocean According to data distribution in be particularly important.

Chinese invention patent CN201610561677.1, publication date is 2016.12.14, discloses a kind of based on SPM and depth The SAR image sorting technique of degree increment SVM.But the method cannot be adapted to oceanographic data, and be unable to reach skill of the invention Art effect.

Therefore, need badly it is a kind of reduce the training time expense and user accesses data response time, solved it is flux matched The oceanographic observation big data location mode based on incremental learning, and had not been reported on this method at present.

The content of the invention

The purpose of the present invention is directed to deficiency of the prior art, there is provided a kind of big number of oceanographic observation based on incremental learning According to location mode.

To achieve the above object, the present invention is adopted the technical scheme that：

A kind of oceanographic observation big data location mode based on incremental learning, the location mode is comprised the following steps：

S1：It is input into increment oceanographic observation data set to be laid out；

S2：Initialization memory capacity；

S3：Calculate the data value that incremental data concentrates data；

S4：All data that incremental data is concentrated are divided；

S5：Incremental data set is trained using Increment Learning Algorithm；

S6：Data after training are laid out；

S7：Increment oceanographic observation data set after output layout；

Wherein, the Increment Learning Algorithm in described step S5 is SVMs Increment Learning Algorithm.

Data value in described step S3 is calculated to be included computational valid time, calculates relevance, calculates region.

Being divided into described step S4 is initially drawn using k-means methods to all data that data are concentrated Point, data set is divided into active region and inactive.

Layout in described step S6 is that the data after training are laid out according to active region and inactive.

The computational methods of described step S3 are comprised the following steps：

S31：Computational valid time

The ageing of oceanographic observation big data is calculated using TF-IDF weighting techniques, its computing formula is as follows：

Wherein, N is the total amount of data of oceanographic observation large data sets, n_iRepresent the data set comprising observation data attribute d Number, tf_iD () represents the frequency that observation data attribute d occurs in data set, W_iD () represents the weights of attribute item d.

S32：Calculate relevance

IfApplication observation data d is represented respectively_kAnd d_mObservation mission, then observe data d_kAnd d_mBetween Degree of association S_ijComputing formula it is as follows：

S33：Calculate region

Using Euclidean distance computational methods calculate in each area of observation coverage between each observation position apart from L_mn, its computing formula is such as Under：

L_mn=√ (x_m-x_n)²+(y_m-y_n)² (3)

Wherein L_mnRepresent the distance between observation station m and observation station n, x_mAnd x_nRepresent observation station m's and observation station n respectively Longitude, y_mAnd y_nThe latitude value of observation station m and observation station n is represented respectively.

Introduce normalization variable, with relative position and it is whole interval in apart from maximum ratio as observation station distance Relating value RL_mn, its computing formula is as follows：

Wherein, RL_mnRepresent the distance between observation station m and observation station n relating value, max { L12, L13, L₁₄,……,L_mn} Expression takes maximum between each distance value.

S34：Calculate data value

For oceanographic observation big data, the frequency used according to it, the download time of data, data consumer it is important The factors such as the production cost of degree and data product, suitably choose the weighting silver of each factor, calculate data value, its calculating Formula is as follows：

V_i(d)=W_i(d)×k₁+S_i(d)×k₂+RL_i(d)×k₃+C (5)

Wherein, V_iD () represents the data value of observation data, W_iD () represents ageing, the S of observation data_iD () represents and sees Survey the relevance of data, RL_iD the region of () with bag observation data, k1 is W_iD the weighted factor of (), k2 is S_iThe weighting of (d) The factor, k3 is RL_iD the weighted factor of (), C represents the penalty factor of data value, by user's attention rate, the data of observation data Collection completes experienced time, the manpower of participation and the experienced link of data production and comprehensively draws.

The workflow of described step S5 is as follows：

S51：The newly-increased ocean big data sample set B of input_i(i=1,2,3 ..., n)；

S52：Judge whether newly-increased sample meets KKT conditions：

S521：If meeting KKT conditions, vector machine (SVM) classification is supported according to KKT conditions, subsequently into step S56；

S522：If not meeting KKT conditions, into step S53；

S53：Judge B_iWhether all on classifying face：

S531：If B_iAll on classifying face, then the sample on class interval is classified as, subsequently into step S56；

S532：If B_iNot all on classifying face, then into step S54；

S54：Judge B_iIt is whether all wrong at the edge of classifying face or former classification：

S541：If B_iIt is all wrong at the edge of classifying face or former classification, then the sample in class interval is classified as, then Into step S56；

S542：If B_iIt is not all errorless at the edge of classifying face or former classification, then into step S55；

S55：According to data value training sample set, i.e., divide sample set using k-means methods；

S56：Output increment sample set.

The invention has the advantages that：

1st, while ensureing classification accuracy rate, the expense of training time and the response time of user accesses data are reduced；

2nd, solved the problems, such as using the Incremental Learning Algorithm of SVMs flux matched；

3rd, it has been effectively compressed the size of sample set and has given up useless sample.

Brief description of the drawings

Accompanying drawing 1 is a kind of flow chart of oceanographic observation big data location mode based on incremental learning of the invention.

Accompanying drawing 2 is a kind of Increment Learning Algorithm of oceanographic observation big data location mode based on incremental learning of the invention Schematic diagram.

Specific embodiment

The specific embodiment that the present invention is provided is elaborated below in conjunction with the accompanying drawings.

The reference and part being related in accompanying drawing are as follows：

Embodiment 1

A kind of reference picture 1, flow of the oceanographic observation big data location mode based on incremental learning of the invention is as follows：

S1：It is input into increment oceanographic observation data set to be laid out；

S2：Initialization memory capacity；

S3：Calculate the data value that incremental data concentrates data；

S4：All data that incremental data is concentrated are divided；

S5：Incremental data set is trained using Increment Learning Algorithm；

S6：Data after training are laid out；

S7：Increment oceanographic observation data set after output layout.

Data value in described step S3 is calculated to be included computational valid time, calculates relevance, calculates region.

Being divided into described step S4 is initially drawn using k-means methods to all data that data are concentrated Point, data set is divided into active region and inactive.

Increment Learning Algorithm in described step S5 is SVMs Increment Learning Algorithm.

Layout in described step S6 is that the data after training are laid out according to active region and inactive.

A kind of advantage of oceanographic observation big data location mode based on incremental learning of the invention is, it is ensured that classification is just While true rate, the expense of training time and the response time of user accesses data are reduced；Using the increment of SVMs Habit algorithm solved the problems, such as flux matched；It has been effectively compressed the size of sample set and has given up useless sample.

Embodiment 2

A kind of data value computational methods of oceanographic observation big data location mode based on incremental learning of the invention are such as Under：

S31：Computational valid time

The ageing of data is one of key factor of influence data distribution.In ocean research field, ocean big data Timeliness sex chromosome mosaicism it is particularly significant, especially for increment type oceanographic observation big data.During Distribution Strategy is designed, data Whether the time span and data access frequency of storage are to differentiate the valuable key factor of data.Data have in the different stages There are different meanings, when data are just stored in data-storage system, the frequency that it is called by user is higher.Over time Increase, this batch data will turn into historical data, the number of times urgency that historical data is called by user relative to firm stored data Reduce sharply small.Thereby it is ensured that data age is a highly important problem.

The ageing of oceanographic observation big data is calculated using TF-IDF weighting techniques, its computing formula is as follows：

Wherein, N is the total amount of data of oceanographic observation large data sets, n_iRepresent the data set comprising observation data attribute d Number, tf_iD () represents the frequency that observation data attribute d occurs in data set, W_iD () represents the weights of attribute item d.

S32：Calculate relevance

The oceanographic data category value of each observation station is various, including longitude, latitude, temperature, humidity, salinity, atmospheric pressure, fluorescent degree Deng, these observation data attribute between there is also certain contact.Therefore, it is laid out to oceanographic observation big data When, it is necessary to consider observe data between the degree of association.

IfApplication observation data d is represented respectively_kAnd d_mObservation mission, then observe data d_kAnd d_mBetween Degree of association S_ijComputing formula it is as follows：

S33：Calculate region

The spatiality of oceanographic data result in the region of oceanographic data.When being observed to each oceanographic data, all Longitude and latitude value where must determine the data, this is also just embodying the feature of oceanographic data spatial coherence.Pass through Observation station is analyzed apart from the degree of association, observation station same number can be as much as possible effectively placed on apart near data According to center, some distant isolated datas are pruned for the later stage, to reduce the memory capacity of data center.

Using Euclidean distance computational methods calculate in each area of observation coverage between each observation position apart from L_mn, its computing formula is such as Under：

L_mn=√ (x_m-x_n)²+(y_m-y_n)² (3)

Wherein L_mnRepresent the distance between observation station m and observation station n, x_mAnd x_nRepresent observation station m's and observation station n respectively Longitude, y_mAnd y_nThe latitude value of observation station m and observation station n is represented respectively.

Introduce normalization variable, with relative position and it is whole interval in apart from maximum ratio as observation station distance Relating value RL_mn, its computing formula is as follows：

Wherein, RL_mnRepresent the distance between observation station m and observation station n relating value, max { L₁₂,L₁₃,L₁₄,……,L_mn} Expression takes maximum between each distance value.

S34：Calculate data value

For oceanographic observation big data, the frequency used according to it, the download time of data, data consumer it is important The factors such as the production cost of degree and data product, suitably choose the weighting silver of each factor, calculate data value, its calculating Formula is as follows：

V_i(d)=W_i(d)×k₁+S_i(d)×k₂+RL_i(d)×k₃+C (5)

Wherein, V_iD () represents the data value of observation data, W_iD () represents ageing, the S of observation data_iD () represents and sees Survey the relevance of data, RL_iD the region of () with bag observation data, k1 is W_iD the weighted factor of (), k2 is S_iThe weighting of (d) The factor, k3 is RL_iD the weighted factor of (), C represents the penalty factor of data value, by user's attention rate, the data of observation data Collection completes experienced time, the manpower of participation and the experienced link of data production and comprehensively draws.

Embodiment 3

Reference picture 2, a kind of incremental learning side of the oceanographic observation big data location mode based on incremental learning of the invention Method is as follows：

S51：The newly-increased ocean big data sample set B of input_i(i=1,2,3 ..., n)；

S52：Judge whether newly-increased sample meets KKT conditions：

S521：If meeting KKT conditions, vector machine (SVM) classification is supported according to KKT conditions, subsequently into step S56；

S522：If not meeting KKT conditions, into step S53；

S53：Judge B_iWhether all on classifying face：

S531：If B_iAll on classifying face, then the sample on class interval is classified as, subsequently into step S56；

S532：If B_iNot all on classifying face, then into step S54；

S54：Judge B_iIt is whether all wrong at the edge of classifying face or former classification：

S541：If B_iIt is all wrong at the edge of classifying face or former classification, then the sample in class interval is classified as, then Into step S56；

S542：If B_iIt is not all errorless at the edge of classifying face or former classification, then into step S55；

S55：According to data value training sample set, i.e., divide sample set using k-means methods；

S56：Output increment sample set.

The above is only the preferred embodiment of the present invention, it is noted that for the ordinary skill people of the art Member, on the premise of the inventive method is not departed from, can also make some improvement and supplement, and these are improved and supplement also should be regarded as Protection scope of the present invention.

Claims (6)

1. a kind of oceanographic observation big data location mode based on incremental learning, it is characterised in that the location mode include with Lower step：

S1：It is input into increment oceanographic observation data set to be laid out；

S2：Initialization memory capacity；

S3：Calculate the data value that incremental data concentrates data；

S4：All data that incremental data is concentrated are divided；

S5：Incremental data set is trained using Increment Learning Algorithm；

S6：Data after training are laid out；

S7：Increment oceanographic observation data set after output layout；

Wherein, the Increment Learning Algorithm in described step S5 is SVMs Increment Learning Algorithm.

2. location mode according to claim 1, it is characterised in that the data value in described step S3 is calculated and included Computational valid time, calculating relevance, calculating region.

3. location mode according to claim 1, it is characterised in that being divided into described step S4 utilizes k- Means methods carry out initial division to all data that data are concentrated, and data set is divided into active region and inactive.

4. location mode according to claim 1, it is characterised in that the layout in described step S6 is to after training Data are laid out according to active region and inactive.

5. location mode according to claim 2, it is characterised in that the computational methods of described step S3 include following step Suddenly：

S31：Computational valid time

The ageing of oceanographic observation big data is calculated using TF-IDF weighting techniques, its computing formula is as follows：

$W_i\left(d\right)=\frac{{\mathrm{ft}}_i\left(d\right)\×\log (\frac{N}{n_i}+0.01)}{\sqrt{\underset{i=1}{\overset{n}{\Σ}}{\left({\mathrm{tf}}_i\right(d\left)\right)}^2\×{\[\log (\frac{N}{n_i}+0.01)\]}^2}}---\left(1\right)$

Wherein, N is the total amount of data of oceanographic observation large data sets, n_iRepresent the number of data sets comprising observation data attribute d, tf_i D () represents the frequency that observation data attribute d occurs in data set, W_iD () represents the weights of attribute item d.

S32：Calculate relevance

If Application observation data d is represented respectively_kAnd d_mObservation mission, then observe data d_kAnd d_mBetween the degree of association S_ijComputing formula it is as follows：

$S_{ij}=count(T_{d_k}\∩T_{d_m})---\left(2\right)$

S33：Calculate region

Using Euclidean distance computational methods calculate in each area of observation coverage between each observation position apart from L_mn, its computing formula is as follows：

L_mn=√ (x_m-x_n)²+(y_m-y_n)² (3)

Wherein L_mnRepresent the distance between observation station m and observation station n, x_mAnd x_nThe longitude of observation station m and observation station n is represented respectively Value, y_mAnd y_nThe latitude value of observation station m and observation station n is represented respectively.

Normalization variable is introduced, is associated as the distance of observation station with the whole interval interior ratio apart from maximum with relative position Value RL_mn, its computing formula is as follows：

${\mathrm{RL}}_{mn}=\frac{L_{mn}}{max\{L_{12},L_{13},L_{14}...L_{mn}\}}---\left(4\right)$

Wherein, RL_mnRepresent the distance between observation station m and observation station n relating value, max { L₁₂,L₁₃,L₁₄,……,L_mnRepresent Maximum is taken between each distance value.

S34：Calculate data value

For oceanographic observation big data, the frequency used according to it, the download time of data, the significance level of data consumer With the factor such as the production cost of data product, the weighting silver of each factor is suitably chosen, calculate data value, its computing formula It is as follows：

V_i(d)=W_i(d)×k₁+S_i(d)×k₂+RL_i(d)×k₃+C (5)

Wherein, V_iD () represents the data value of observation data, W_iD () represents ageing, the S of observation data_iD () represents observation number According to relevance, RL_iD the region of () with bag observation data, k1 is W_iD the weighted factor of (), k2 is S_iThe weighted factor of (d), K3 is RL_iD the weighted factor of (), C represents the penalty factor of data value, by user's attention rate, the data acquisition of observation data Experienced time, the manpower of participation and the experienced link of data production is completed comprehensively to draw.

6. location mode according to claim 1, it is characterised in that the workflow of described step S5 is as follows：

S51：The newly-increased ocean big data sample set B of input_i(i=1,2,3 ..., n)；

S52：Judge whether newly-increased sample meets KKT conditions：

S521：If meeting KKT conditions, vector machine (SVM) classification is supported according to KKT conditions, subsequently into step S56；

S522：If not meeting KKT conditions, into step S53；

S53：Judge B_iWhether all on classifying face：

S531：If B_iAll on classifying face, then the sample on class interval is classified as, subsequently into step S56；

S532：If B_iNot all on classifying face, then into step S54；

S54：Judge B_iIt is whether all wrong at the edge of classifying face or former classification：

S541：If B_iIt is all wrong at the edge of classifying face or former classification, then the sample in class interval is classified as, subsequently into Step S56；

S542：If B_iIt is not all errorless at the edge of classifying face or former classification, then into step S55；

S55：According to data value training sample set, i.e., divide sample set using k-means methods；

S56：Output increment sample set.