CN108268891A - A kind of data processing method - Google Patents
A kind of data processing method Download PDFInfo
- Publication number
- CN108268891A CN108268891A CN201711479249.5A CN201711479249A CN108268891A CN 108268891 A CN108268891 A CN 108268891A CN 201711479249 A CN201711479249 A CN 201711479249A CN 108268891 A CN108268891 A CN 108268891A
- Authority
- CN
- China
- Prior art keywords
- data
- annexable
- calculating
- classification
- cluster centre
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/23—Clustering techniques
- G06F18/232—Non-hierarchical techniques
- G06F18/2321—Non-hierarchical techniques using statistics or function optimisation, e.g. modelling of probability density functions
- G06F18/23213—Non-hierarchical techniques using statistics or function optimisation, e.g. modelling of probability density functions with fixed number of clusters, e.g. K-means clustering
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
Landscapes
- Engineering & Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Artificial Intelligence (AREA)
- Bioinformatics & Computational Biology (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Evolutionary Biology (AREA)
- Evolutionary Computation (AREA)
- Bioinformatics & Cheminformatics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Probability & Statistics with Applications (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a kind of data processing methods, pass through the collected sampled data of sensor passage interface data collection station；The sampled data is read, and according to multiple default cluster centres, classifies to the processing data, obtains post-classification comparison data；According to the post-classification comparison data, multiple annexable calculating tasks are established；The annexable calculating task is calculated, and operation is merged to result of calculation using multiple computational threads；The default cluster centre is modified and preserved according to the result of calculation after merging；And according to the default cluster centre, revised default cluster centre and number of operations is corrected, determine data clusters handling result；The corresponding drawing window of the parser of generation and selection, and the analysis result data is shown in the drawing window.Using the embodiment of the present invention, the validity of data processing is improved.
Description
Technical field
The present invention relates to technical field of data processing more particularly to a kind of data processing methods.
Background technology
Video monitoring is had been widely used in various public arenas and Private Lounge.The video monitoring system of existing market mainstream
Camera, client and server including acquiring video data.Wherein, client receives the video counts of camera acquisition
According to rear, generally it will can all be uploaded to server.Since the video data volume is very big, so being regarded by user end to server upload
Frequency according to when, need higher network bandwidth, thus, this method for uploading total data is to the Netowrk tape of video monitoring system
Wide configuration requirement is higher.Moreover, server will preserve all data of upload, since video data is very big, this is just needed
Server needs very big memory capacity to store video data.Thus, video monitoring system of the prior art due to
It needs all videos data to be uploaded to the higher network bandwidth configuration of server needs, and since server will store upload
All video datas, it is also higher to the requirement of the storage capacity configuration of server.
Invention content
The embodiment of the present invention is designed to provide a kind of data processing method, and to solve, this baud rate is incompatible to ask
Topic, realizes the normal communication between different baud rate equipment, also realizes that more set systems share the system integration of communication bus.Tool
Body technique scheme is as follows：
In order to achieve the above objectives, an embodiment of the present invention provides a kind of data processing method, the method includes the steps：
Pass through the collected sampled data of sensor passage interface data collection station；
The sampled data is read, and according to multiple default cluster centres, is classified to the processing data, after obtaining classification
Handle data；
According to the post-classification comparison data, multiple annexable calculating tasks are established；
The annexable calculating task is calculated, and operation is merged to result of calculation using multiple computational threads；
The default cluster centre is modified and preserved according to the result of calculation after merging；And
According to the default cluster centre, revised default cluster centre and number of operations is corrected, is determined at data clusters
Manage result；
The corresponding drawing window of the parser of generation and selection, and the analysis knot is shown in the drawing window
Fruit data.
The present invention a kind of realization method in, it is described obtain it is described processing data multiple default cluster centres the step of wrap
It includes：
Judge whether the corresponding Saved Presets cluster centre of the processing data, directly acquired if existing as described in advance
If cluster centre is such as not present, then the data in the processing data are randomly choosed as the default cluster centre.
It is described and according to multiple default cluster centres in a kind of realization method of the present invention, the processing data are carried out
The step of classifying, obtaining post-classification comparison data includes：
The position of every data in the processing data and the distance of all default cluster centres are calculated, it will be with the data
Classification point of the nearest default cluster centre as the data；And according to it is described classification point to it is described processing data into
Row classification, obtains post-classification comparison data.
It is described according to post-classification comparison data in a kind of realization method of the present invention, it establishes multiple annexable calculate and appoints
The step of business, includes：Post-classification comparison data are divided into multiple calculating data portions；And by each calculating data portion
Classification and computation rule in the processing data, establish multiple annexable calculating tasks.
It is described that the annexable calculating task is carried out using multiple computational threads in a kind of realization method of the present invention
The step of calculating, includes：By the storage location of the data of the quantity and annexable calculating task of the computational threads,
The data of each annexable calculating task are grouped；It is and other described to respective sets using the computational threads
Annexable calculating task is calculated.
Using a kind of data processing method provided in an embodiment of the present invention, acquired by sensor passage interface data
The collected sampled data of terminal；And according to multiple default cluster centres, classify to the processing data, after obtaining classification
Handle data；Annexable calculate is appointed using multiple computational threads by multiple annexable calculating tasks of foundation again
Business is calculated, and operation is merged to result of calculation；According to the result of calculation after merging to the default cluster centre into
Row is corrected and is preserved；And according to the default cluster centre, revised default cluster centre and number of operations is corrected,
Determine data clusters handling result.Reduce the mistake in data handling procedure, by carrying out sorted calculating and calculating
Union operation and cluster afterwards is corrected, and the accuracy of data processing is improved, so as to further improve the validity of data processing.
Description of the drawings
In order to illustrate more clearly about the embodiment of the present invention or technical scheme of the prior art, to embodiment or will show below
There is attached drawing needed in technology description to be briefly described, it should be apparent that, the accompanying drawings in the following description is only this
Some embodiments of invention, for those of ordinary skill in the art, without creative efforts, can be with
Other attached drawings are obtained according to these attached drawings.
Fig. 1 is the flow diagram of the data processing method of the embodiment of the present invention.
Specific embodiment
Below in conjunction with the attached drawing in the embodiment of the present invention, the technical solution in the embodiment of the present invention is carried out clear, complete
Site preparation describes, it is clear that described embodiment is only part of the embodiment of the present invention, instead of all the embodiments.It is based on
Embodiment in the present invention, those of ordinary skill in the art are obtained every other without making creative work
Embodiment shall fall within the protection scope of the present invention.
To solve prior art problem, an embodiment of the present invention provides a kind of data processing methods, and the method includes steps
Suddenly：
S101 passes through the collected sampled data of sensor passage interface data collection station.
In the embodiment of the present invention, pass through the collected sampled data of sensor passage interface data collection station.This
In inventive embodiments, data collection station can be sensor, such as wireless sensor.
S102, reads the sampled data, and according to multiple default cluster centres, classifies to the processing data,
Obtain post-classification comparison data.
It is understood that the present invention is applied in data processor, executive agent can be data processor.It is read
Sampled data are taken, and are clustered using preset clustering algorithm, it is to be understood that cluster（Cluster）Analysis is
If by dry model（Pattern）Composition, in general, pattern is a measurement（Measurement）Vector or multidimensional it is empty
Between in a point, the result after being clustered using clustering algorithm.Sampled data is divided into several classifications.
S103 according to the post-classification comparison data, establishes multiple annexable calculating tasks.
It, can be according to inhomogeneous data using different computational methods, so as to be formed not after data are clustered
Since the data of each class after cluster have certain similitude, meter is for each for same calculating task
For calculation task, the computational efficiency of cluster can be improved.Therefore, the merging calculating of the task can be established, can as be merged
Calculating task.
S104 calculates the annexable calculating task using multiple computational threads, and result of calculation is carried out
Union operation.
It can merge task using different threads pair to calculate, a computational threads specifically may be used can to one
The calculating task of merging is calculated, and can also be used multiple computational threads that can merge task to one and be calculated, can also
At least one annexable calculating task is calculated using a computational threads.And the result after calculating is merged into behaviour
Make.Specific thread and the quantity correspondence embodiment of the present invention of annexable calculating task do not limit.
S105 is modified and preserves to the default cluster centre according to the result of calculation after merging；And according to
The default cluster centre, revised default cluster centre and amendment number of operations, determine data clusters handling result.
Result of calculation after merging can reflect the classification accuracy of clustering algorithm used in cluster centre, when classification
It needs to be modified algorithm when accuracy rate is too low, which is the step of amendment is repeated.
S106, the corresponding drawing window of the parser of generation and selection, and shown in the drawing window
The analysis result data.
In the embodiment of the present invention, cluster is subjected to display analysis result data in corresponding drawing window.
Therefore, using a kind of data processing method provided in an embodiment of the present invention, pass through sensor passage interface number
According to the collected sampled data of acquisition terminal；And according to multiple default cluster centres, classify to the processing data, obtain
Post-classification comparison data；Multiple computational threads are used to described annexable by multiple annexable calculating tasks of foundation again
Calculating task is calculated, and operation is merged to result of calculation；According to the result of calculation after merging to the default cluster
Center is modified and preserves；And according to the default cluster centre, revised default cluster centre and correct behaviour
Make number, determine data clusters handling result.Reduce the mistake in data handling procedure, by carrying out sorted calculating,
And the union operation after calculating is corrected with cluster, the accuracy of data processing is improved, so as to further improve data processing
Validity.
The present invention a kind of realization method in, it is described obtain it is described processing data multiple default cluster centres the step of wrap
It includes：Judge whether the corresponding Saved Presets cluster centre of the processing data, directly acquired if existing as described in advance
If cluster centre is such as not present, then the data in the processing data are randomly choosed as the default cluster centre.
It is described and according to multiple default cluster centres in a kind of realization method of the present invention, the processing data are carried out
The step of classifying, obtaining post-classification comparison data includes：Calculate the position of every data in the processing data and all
The distance of default cluster centre, using the default cluster centre nearest with the data as the classification point of the data；With
And classified according to the classification point to the processing data, obtain post-classification comparison data.
Illustratively, KMeans algorithms are most common clustering algorithms, and main thought is:To defining K value and K initial classes
In the case of cluster central point, each point (that is, data record) is assigned to the class cluster representated by the class cluster central point of nearest neighbours
In, after all the points are assigned, the central point (being averaged) of such cluster is recalculated according to all the points in a class cluster,
Then iteration is allocated a little and the step of update class cluster central point again, until class cluster central point varies less, Huo Zheda
To the iterations specified.
It is described according to post-classification comparison data in a kind of realization method of the present invention, it establishes multiple annexable calculate and appoints
The step of business, includes：Post-classification comparison data are divided into multiple calculating data portions；And by each calculating data portion
Classification and computation rule in the processing data, establish multiple annexable calculating tasks.
In the embodiment of the present invention, by the way that post-classification comparison data are divided into multiple calculating data portions；And by each institute
The classification and computation rule for calculating data portion in the processing data are stated, multiple annexable calculate is established and appoints
Business, the calculating of data is carried out by classification, and obtains annexable calculating task, further improves the cluster accuracy rate of data.
It is described that the annexable calculating task is carried out using multiple computational threads in a kind of realization method of the present invention
The step of calculating, includes：By the storage location of the data of the quantity and annexable calculating task of the computational threads,
The data of each annexable calculating task are grouped；It is and other described to respective sets using the computational threads
Annexable calculating task is calculated.
The foregoing is merely illustrative of the preferred embodiments of the present invention, is not intended to limit the scope of the present invention.It is all
Any modification, equivalent replacement, improvement and so within the spirit and principles in the present invention, are all contained in protection scope of the present invention
It is interior.
Claims (5)
1. a kind of data processing method, which is characterized in that the method includes the steps：
Pass through the collected sampled data of sensor passage interface data collection station；
The sampled data is read, and according to multiple default cluster centres, is classified to the processing data, after obtaining classification
Handle data；
According to the post-classification comparison data, multiple annexable calculating tasks are established；
The annexable calculating task is calculated, and operation is merged to result of calculation using multiple computational threads；
The default cluster centre is modified and preserved according to the result of calculation after merging；And according to described default poly-
Class center, revised default cluster centre and amendment number of operations, determine data clusters handling result；
The corresponding drawing window of the parser of generation and selection, and the analysis knot is shown in the drawing window
Fruit data.
2. data processing method according to claim 1, which is characterized in that described to obtain the multiple pre- of the processing data
If the step of cluster centre, includes：
Judge whether the corresponding Saved Presets cluster centre of the processing data, directly acquired if existing as described in advance
If cluster centre is such as not present, then the data in the processing data are randomly choosed as the default cluster centre.
3. data processing method according to claim 2, which is characterized in that it is described and according to multiple default cluster centres,
The step of classifying to the processing data, obtain post-classification comparison data includes：
The position of every data in the processing data and the distance of all default cluster centres are calculated, it will be with the data
Classification point of the nearest default cluster centre as the data；And according to it is described classification point to it is described processing data into
Row classification, obtains post-classification comparison data.
4. data processing method according to claim 2, which is characterized in that it is described according to post-classification comparison data, it establishes
The step of multiple annexable calculating tasks, includes：Post-classification comparison data are divided into multiple calculating data portions；And by every
A classification and computation rule for calculating data portion in the processing data, establishes multiple annexable calculating
Task.
5. according to the data processing method any in Claims 1-4, which is characterized in that described to use multiple calculating lines
The step of journey calculates the annexable calculating task includes：By the computational threads quantity and described merge
Calculating task data storage location, the data of each annexable calculating task are grouped；And it uses
To respective sets, other annexable calculating task calculates the computational threads.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711479249.5A CN108268891A (en) | 2017-12-29 | 2017-12-29 | A kind of data processing method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711479249.5A CN108268891A (en) | 2017-12-29 | 2017-12-29 | A kind of data processing method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN108268891A true CN108268891A (en) | 2018-07-10 |
Family
ID=62773104
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201711479249.5A Pending CN108268891A (en) | 2017-12-29 | 2017-12-29 | A kind of data processing method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108268891A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110717517A (en) * | 2019-09-06 | 2020-01-21 | 中国平安财产保险股份有限公司 | Intelligent multithreading clustering method and device and computer readable storage medium |
| WO2020113470A1 (en) * | 2018-12-05 | 2020-06-11 | 深圳大学 | Data block division method and apparatus, and terminal device |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101853491A (en) * | 2010-04-30 | 2010-10-06 | 西安电子科技大学 | SAR (Synthetic Aperture Radar) image segmentation method based on parallel sparse spectral clustering |
| CN102364466A (en) * | 2011-10-08 | 2012-02-29 | 王浩 | Method for dynamically excavating opportunity information based on human-computer interaction |
| CN104360824A (en) * | 2014-11-10 | 2015-02-18 | 北京奇虎科技有限公司 | Data merging method and device |
| CN104578120A (en) * | 2014-12-11 | 2015-04-29 | 国网重庆市电力公司经济技术研究院 | Optimal configuration method for distributed energy storage system |
| CN106127662A (en) * | 2016-06-23 | 2016-11-16 | 福州大学 | A kind of system of selection of the K means initial cluster center for taxi track data |
| CN106295670A (en) * | 2015-06-11 | 2017-01-04 | 腾讯科技（深圳）有限公司 | Data processing method and data processing equipment |
| CN106529608A (en) * | 2016-12-31 | 2017-03-22 | 中国科学技术大学 | Comprehensive stationing system for exhaust gas remote-measuring equipment of motor vehicle |
-
2017
- 2017-12-29 CN CN201711479249.5A patent/CN108268891A/en active Pending
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101853491A (en) * | 2010-04-30 | 2010-10-06 | 西安电子科技大学 | SAR (Synthetic Aperture Radar) image segmentation method based on parallel sparse spectral clustering |
| CN102364466A (en) * | 2011-10-08 | 2012-02-29 | 王浩 | Method for dynamically excavating opportunity information based on human-computer interaction |
| CN104360824A (en) * | 2014-11-10 | 2015-02-18 | 北京奇虎科技有限公司 | Data merging method and device |
| CN104578120A (en) * | 2014-12-11 | 2015-04-29 | 国网重庆市电力公司经济技术研究院 | Optimal configuration method for distributed energy storage system |
| CN106295670A (en) * | 2015-06-11 | 2017-01-04 | 腾讯科技（深圳）有限公司 | Data processing method and data processing equipment |
| CN106127662A (en) * | 2016-06-23 | 2016-11-16 | 福州大学 | A kind of system of selection of the K means initial cluster center for taxi track data |
| CN106529608A (en) * | 2016-12-31 | 2017-03-22 | 中国科学技术大学 | Comprehensive stationing system for exhaust gas remote-measuring equipment of motor vehicle |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2020113470A1 (en) * | 2018-12-05 | 2020-06-11 | 深圳大学 | Data block division method and apparatus, and terminal device |
| CN110717517A (en) * | 2019-09-06 | 2020-01-21 | 中国平安财产保险股份有限公司 | Intelligent multithreading clustering method and device and computer readable storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN104348747B (en) | The method and system of traffic monitoring in MPLS VPNs | |
| CN106294511B (en) | A kind of storage method and device of Hadoop distributed file system | |
| CN112543465A (en) | Abnormity detection method, abnormity detection device, terminal and storage medium | |
| CN107656807A (en) | The automatic elastic telescopic method and device of a kind of virtual resource | |
| CN108268891A (en) | A kind of data processing method | |
| CN109918498A (en) | A kind of problem storage method and device | |
| CN117459418B (en) | Real-time data acquisition and storage method and system | |
| CN114817425B (en) | Method, device and equipment for classifying cold and hot data and readable storage medium | |
| CN108696369A (en) | A kind of warning information processing equipment and method | |
| CN109976901A (en) | A kind of resource regulating method, device, server and readable storage medium storing program for executing | |
| CN108924057A (en) | The port flow intelligence control system of system on a kind of cloud | |
| CN106817314A (en) | Big data acquisition method, device and system | |
| CN104391879A (en) | Method and device for hierarchical clustering | |
| CN107181616A (en) | A kind of method and system for monitoring performance of storage system data | |
| CN108647727A (en) | Unbalanced data classification lack sampling method, apparatus, equipment and medium | |
| CN106059816B (en) | Transfer storage facility site selecting method and system | |
| CN113315679B (en) | Link packet loss rate inference method and system based on network resource characteristics | |
| CN109952743A (en) | System and method for low memory and low discharge expense big flow object detection | |
| CN110851282B (en) | Distributed data calculation method and system based on memory grid | |
| CN107483634A (en) | A kind of load-balancing method and system | |
| CN116166444A (en) | Collaborative reasoning method oriented to deep learning hierarchical model | |
| CN106131238A (en) | The sorting technique of IP address and device | |
| CN117875746A (en) | Method and device for determining index threshold, storage medium and electronic device | |
| CN115951123A (en) | Electric energy metering method and system based on wireless communication | |
| WO2019011800A1 (en) | Reliability of network mappings |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20180710 |