CN114219023A - Data clustering method and device, electronic equipment and readable storage medium - Google Patents
Data clustering method and device, electronic equipment and readable storage medium Download PDFInfo
- Publication number
- CN114219023A CN114219023A CN202111525958.9A CN202111525958A CN114219023A CN 114219023 A CN114219023 A CN 114219023A CN 202111525958 A CN202111525958 A CN 202111525958A CN 114219023 A CN114219023 A CN 114219023A
- Authority
- CN
- China
- Prior art keywords
- data
- point
- centroid
- clustering
- points
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 48
- 230000011218 segmentation Effects 0.000 claims abstract description 56
- 238000012545 processing Methods 0.000 claims abstract description 30
- 238000004364 calculation method Methods 0.000 claims description 13
- 238000004590 computer program Methods 0.000 claims description 12
- 238000013507 mapping Methods 0.000 claims description 7
- 230000008569 process Effects 0.000 claims description 5
- 238000005516 engineering process Methods 0.000 abstract description 11
- 238000013473 artificial intelligence Methods 0.000 abstract description 7
- 238000004422 calculation algorithm Methods 0.000 description 22
- 238000004891 communication Methods 0.000 description 7
- 230000006870 function Effects 0.000 description 7
- 238000000638 solvent extraction Methods 0.000 description 7
- 238000007726 management method Methods 0.000 description 5
- 238000010586 diagram Methods 0.000 description 4
- 230000000694 effects Effects 0.000 description 4
- 238000000605 extraction Methods 0.000 description 4
- 239000004973 liquid crystal related substance Substances 0.000 description 4
- 238000005259 measurement Methods 0.000 description 4
- 238000005192 partition Methods 0.000 description 4
- 238000011156 evaluation Methods 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 239000002245 particle Substances 0.000 description 2
- 238000012216 screening Methods 0.000 description 2
- 230000002776 aggregation Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000003058 natural language processing Methods 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 238000012163 sequencing technique Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/23—Clustering techniques
- G06F18/232—Non-hierarchical techniques
- G06F18/2321—Non-hierarchical techniques using statistics or function optimisation, e.g. modelling of probability density functions
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q40/00—Finance; Insurance; Tax strategies; Processing of corporate or income taxes
- G06Q40/08—Insurance
Landscapes
- Engineering & Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Business, Economics & Management (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Finance (AREA)
- Accounting & Taxation (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Evolutionary Computation (AREA)
- General Engineering & Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Bioinformatics & Computational Biology (AREA)
- Artificial Intelligence (AREA)
- Evolutionary Biology (AREA)
- Probability & Statistics with Applications (AREA)
- Development Economics (AREA)
- Economics (AREA)
- Marketing (AREA)
- Strategic Management (AREA)
- Technology Law (AREA)
- General Business, Economics & Management (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention relates to artificial intelligence, and discloses a data clustering method, which comprises the following steps: the method comprises the steps of obtaining an original service data set, carrying out region division on the original service data set to obtain an original region data set, carrying out grid division on the original region data set, calculating centroid points of data in a grid to obtain a centroid point set, carrying out clustering processing on the centroid point set to obtain a clustering point set, and carrying out category cluster segmentation on the clustering point set based on a dichotomy to obtain a clustering result. Furthermore, the invention also relates to a blockchain technology, and the clustering result can be stored in a node of the blockchain. The invention also provides a data clustering method and device, electronic equipment and a computer readable storage medium. The invention can solve the problem of low clustering efficiency.
Description
Technical Field
The invention relates to the technical field of artificial intelligence, in particular to a data clustering method, a data clustering device, electronic equipment and a computer readable storage medium.
Background
With the development of science and technology, more and more data needs to be analyzed in combination with geographical locations. For example, in business operation, insurance companies accumulate a large amount of insurance data, and the insurance data can be mapped into a geographic space through addresses, so as to form risk zone maps of various subjects, such as a flooding risk zone map. In the process of extracting the high-risk area from a large number of points, the points are clustered, the spatially adjacent points are classified into a cluster, and then the area is extracted. In the process, as the element of risk point distribution density needs to be considered, a density clustering algorithm is used for clustering the punctiform risk data. In the existing density clustering algorithm, the DBSCAN algorithm is simple in concept and wide in application, but is sensitive to parameters, cannot be self-adaptive to point groups with large density spatial distribution difference, and is unreasonable in generated result for extracting a task target of dense clusters in a risk division; the OPTIC algorithm is improved based on DBSCAN, an reachable distance concept is provided, a clustering result is provided in a reachable distance sequence mode, the problem that the DBSCAN is too sensitive to parameters is solved, the clustering of self-adaptive density is achieved, however, the operation speed of the algorithm is limited based on a reachable distance sequencing mode, and the time spent on clustering operation of a large number of points is long. Therefore, a method for improving data clustering efficiency is needed.
Disclosure of Invention
The invention provides a data clustering method, a data clustering device, data clustering equipment and a storage medium, and mainly aims to solve the problem of low clustering efficiency.
In order to achieve the above object, the present invention provides a data clustering method, including:
acquiring an original service data set, and performing region division on the original service data set to obtain an original region data set;
carrying out grid division on the original region data set, and calculating centroid points of data in grids to obtain a centroid point set;
clustering the centroid point set to obtain a clustering point set;
and performing category cluster segmentation on the clustering point set based on a dichotomy to obtain a clustering result.
Optionally, the performing region division on the original service data set to obtain an original region data set includes:
mapping the service data in the original service data set into service point data in a target area according to the geographical position of the service data in the original service data set;
and summarizing the service point data of all the target areas to obtain the original area data set.
Optionally, the mesh partitioning of the original region data set includes:
constructing a graticule by using preset longitude intervals and latitude intervals;
and performing grid division on the service point data in the target area by using the transit network, and summarizing all grid areas containing the service point data.
Optionally, the calculating centroid points of data in the mesh to obtain a set of centroid points includes:
selecting any grid area as a target grid, and calculating the centroid points of all service point data in the target grid by using a preset centroid calculation formula;
and assigning the number of the service data points in the target grid to the centroid point as a centroid attribute, and summarizing the assigned centroid points of all grid areas to obtain the centroid point set.
Optionally, the clustering the centroid point set to obtain a cluster point set includes:
selecting any center of mass point in the center of mass point set as a target center of mass point;
traversing centroid points which are not selected as target centroid points in the centroid point set, and calculating whether the sum of the traversed centroid points and the centroid attributes of the target centroid points is larger than a preset range threshold value or not;
if the sum of the centroid attributes of the centroid point and the target centroid point is not larger than a preset range threshold, reselecting the target centroid point, and returning to the step of traversing the centroid points which are not selected as the target centroid points in the centroid point set;
if the sum of the centroid attributes of the centroid point and the target centroid point is larger than a preset range threshold, determining the target centroid point as a core data point, determining the distance between the centroid point and the target centroid point as a core distance, and calculating the reachable distance of the core data point according to the core distance;
and when all the centroid points are selected as target centroid points, summarizing all the core data points, the core distances of the core data points and the reachable distances of the core data points to obtain the cluster point set.
Optionally, the performing category cluster segmentation on the clustering point set based on the dichotomy to obtain a clustering result includes:
selecting the data point with the largest reachable distance in the cluster point set as a segmentation point, and segmenting the cluster point set into two category clusters according to the segmentation point;
repeatedly selecting the data point with the largest reachable distance from the two category clusters as a segmentation point to perform iterative segmentation on the two category clusters until a preset iteration condition is met, and stopping segmentation;
and obtaining the clustering result according to all the segmentation points.
Optionally, the obtaining the clustering result according to all the segmentation points includes:
determining a data point in the middle of the segmentation points in the clustering point set as a clustering center;
and mapping the clustering center back to the grid area to obtain the clustering result.
In order to solve the above problem, the present invention also provides a data clustering apparatus, including:
the system comprises a region division module, a data acquisition module and a data processing module, wherein the region division module is used for acquiring an original service data set and carrying out region division on the original service data set to obtain an original region data set;
the grid division module is used for carrying out grid division on the original region data set and calculating centroid points of data in grids to obtain a centroid point set;
the centroid clustering module is used for clustering the centroid point set to obtain a clustering point set;
and the category cluster segmentation module is used for performing category cluster segmentation on the clustering point set based on a dichotomy to obtain a clustering result.
In order to solve the above problem, the present invention also provides an electronic device, including:
a memory storing at least one computer program; and
and the processor executes the computer program stored in the memory to realize the data clustering method.
In order to solve the above problem, the present invention also provides a computer-readable storage medium, in which at least one computer program is stored, and the at least one computer program is executed by a processor in an electronic device to implement the data clustering method described above.
According to the invention, the original service data set is subjected to region division to obtain the original region data set, the original region data set is subjected to grid division, a large amount of data is divided into different grid regions, and the data processing efficiency is improved. Moreover, by calculating the centroid points of the data in the grid, the data can be converted into a relatively small centroid point set, and the data volume is further reduced, so that the clustering processing is performed on the centroid point set, and the clustering efficiency can be improved. Meanwhile, the clustering point set is subjected to category cluster segmentation through a dichotomy, clusters formed by extracting mass center points after segmentation points are obtained are expanded, and the clustering method has density self-adaptability and enables clustering results to be more accurate. Therefore, the data clustering method, the data clustering device, the electronic equipment and the computer readable storage medium provided by the invention can solve the problem of low clustering efficiency.
Drawings
Fig. 1 is a schematic flow chart of a data clustering method according to an embodiment of the present invention;
FIG. 2 is a functional block diagram of a data clustering device according to an embodiment of the present invention;
fig. 3 is a schematic structural diagram of an electronic device for implementing the data clustering method according to an embodiment of the present invention.
The implementation, functional features and advantages of the objects of the present invention will be further explained with reference to the accompanying drawings.
Detailed Description
It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
The embodiment of the application provides a data clustering method. The execution subject of the data clustering method includes, but is not limited to, at least one of electronic devices such as a server and a terminal, which can be configured to execute the method provided by the embodiments of the present application. In other words, the data clustering method may be performed by software or hardware installed in a terminal device or a server device, and the software may be a blockchain platform. The server includes but is not limited to: a single server, a server cluster, a cloud server or a cloud server cluster, and the like. The server may be an independent server, or may be a cloud server that provides basic cloud computing services such as a cloud service, a cloud database, cloud computing, a cloud function, cloud storage, a Network service, cloud communication, a middleware service, a domain name service, a security service, a Content Delivery Network (CDN), a big data and artificial intelligence platform, and the like.
Fig. 1 is a schematic flow chart of a data clustering method according to an embodiment of the present invention.
In this embodiment, the data clustering method includes:
s1, acquiring an original service data set, and performing region division on the original service data set to obtain an original region data set.
In the embodiment of the present invention, the original service data set may be service data related to a geographic area, for example, in the insurance field, the original service data set may be insurance data of different regions.
Specifically, the performing region division on the original service data set to obtain an original region data set includes:
mapping the service data in the original service data set into service point data in a target area according to the geographical position of the service data in the original service data set;
and summarizing the service point data of all the target areas to obtain the original area data set.
In an optional embodiment of the present invention, taking flooding insurance in the insurance field as an example, according to an accident location of an insurance order, the order is mapped to an insurance data point in a map.
In the embodiment of the invention, when the dense clusters are extracted based on the risk points, the distance measurement between the points uses a geographical distance concept, namely, the clustering based on the spatial distance measurement. When a large amount of point data exists, the point data is directly partitioned in a target area in space, and then algorithm clustering is respectively applied to the point data in each partition in a parallel mode, so that the processing efficiency can be greatly improved. The partitioning mode is usually performed based on administrative divisions, and target areas such as provinces, cities or districts can be selected as partitioning bases by combining with evaluation granularity required by services, so that a large amount of original data are partitioned into different target areas, and the data processing efficiency is improved.
And S2, carrying out grid division on the original area data set, and calculating the centroid point of the data in the grid to obtain a centroid point set.
In the embodiment of the invention, after data partitioning is carried out, in order to further improve the data processing efficiency, the data points in the target area are further subjected to grid division.
In detail, the meshing the original region data set includes:
constructing a graticule by using preset longitude intervals and latitude intervals;
and performing grid division on the service point data in the target area by using the transit network, and summarizing all grid areas containing the service point data.
In the embodiment of the invention, because the data points are not uniformly distributed in the target area, no data point exists in some areas, and the clustering effect is influenced, the data processing efficiency can be improved by screening the data by using the grids.
Specifically, the calculating centroid points of data in the mesh to obtain a centroid point set includes:
selecting any grid area as a target grid, and calculating the centroid points of all service point data in the target grid by using a preset centroid calculation formula;
and assigning the number of the service data points in the target grid to the centroid point as a centroid attribute, and summarizing the assigned centroid points of all grid areas to obtain the centroid point set.
In the embodiment of the invention, a data distribution area is subjected to grid division according to a fixed longitude and latitude interval in a way of dividing a longitude and latitude network in a partition of a target area, after division, a center of mass point is obtained by calculation according to all risk data points in each grid area with risk data points, the total number of the risk data points in each grid area is given to the center of mass point as an attribute value nreports, and after processing, a point set P consisting of a series of center of mass points is obtained, wherein each center of mass point in the point set P is a characteristic point of all risk data points in the corresponding grid area.
In an optional embodiment of the present invention, the calculating the centroid points of all service point data in the target grid by using a preset centroid calculation formula includes:
calculating the centroid points of all service point data in the target grid by using the following centroid calculation formula:
wherein x is the abscissa of the centroid point, y is the ordinate of the centroid point, and xiIs the abscissa, y, of the ith business data pointiIs the ordinate of the ith traffic data point.
In the embodiment of the invention, a large number of service data points can be converted into a relatively small centroid point set through grid division, the data volume is greatly reduced, and the clustering effect is improved.
And S3, clustering the centroid point set to obtain a clustering point set.
In the embodiment of the present invention, since the clustering object is converted into the centroid point set, an improved OPTICS algorithm may be used for clustering, that is: for any center point piContinuously searching the nearest centroid point, and when finding the jth centroid point pjWhen the sum of the nreports attribute value and the central centroid point is greater than MinPts (preset range parameter), then piHas a core distance of piAnd pjDistance between p and piIs consistent with the definition of the original OPTICS algorithm.
In detail, the clustering the centroid point set to obtain a cluster point set includes:
selecting any center of mass point in the center of mass point set as a target center of mass point;
traversing centroid points which are not selected as target centroid points in the centroid point set, and calculating whether the sum of the traversed centroid points and the centroid attributes of the target centroid points is larger than a preset range threshold value or not;
if the sum of the centroid attributes of the centroid point and the target centroid point is not larger than a preset range threshold, reselecting the target centroid point, and returning to the step of traversing the centroid points which are not selected as the target centroid points in the centroid point set;
if the sum of the centroid attributes of the centroid point and the target centroid point is larger than a preset range threshold, determining the target centroid point as a core data point, determining the distance between the centroid point and the target centroid point as a core distance, and calculating the reachable distance of the core data point according to the core distance;
and when all the centroid points are selected as target centroid points, summarizing all the core data points, the core distances of the core data points and the reachable distances of the core data points to obtain the cluster point set.
In the embodiment of the invention, clustering is performed through an improved OPTICS algorithm, and each data point has two attributes, namely: core distance and reach distance.
In an optional embodiment of the present invention, the calculating the reachable distance of the core data point according to the core distance includes:
calculating the reachable distance of the core data point using the following formula:
rd(y,x)＝min{ε:y∈Nε(x) And | Nε(x)|≥MinPts}
Wherein x is a core data point, y is a traversed centroid point, epsilon is a preset field, and N isε(x) MinPts is a predetermined range parameter for the subsamples obtained according to the predetermined field.
In the embodiment of the invention, the clustering point set output by the OPTIC algorithm comprises three ordered arrangements, a core data point sequence, a core distance sequence and an reachable distance sequence.
And S4, performing category cluster segmentation on the clustering point set based on a dichotomy to obtain a clustering result.
In the embodiment of the invention, the dichotomy refers to that for a clustering point set, each processing is only divided into two parts, and repeated iteration is carried out for segmentation, so that the data processing efficiency can be improved.
In detail, the performing category cluster segmentation on the clustering point set based on the dichotomy to obtain a clustering result includes:
selecting the data point with the largest reachable distance in the cluster point set as a segmentation point, and segmenting the cluster point set into two category clusters according to the segmentation point;
repeatedly selecting the data point with the largest reachable distance from the two category clusters as a segmentation point to perform iterative segmentation on the two category clusters until a preset iteration condition is met, and stopping segmentation;
and obtaining the clustering result according to all the segmentation points.
In the embodiment of the present invention, for example, after improved OPTICS clustering is performed, an reachable distance sequence S having the same length as the centroid point set P is obtained, at this time, a cluster is extracted, and a segmentation boundary between clusters is selected based on S, which is specifically as follows: 1. finding a maximum value point of the reachable distance sequence S, taking the point as a dividing point, dividing the sequence into a left area S _ left and a right area S _ right, judging whether the number of sample points of the S _ left or the S _ right is less than a preset threshold value or is all null values, and if so, stopping iteration; 2. and (4) repeating the step (1) for the S _ left and the S _ right respectively until iteration is stopped, and finding all the segmentation points.
In detail, the obtaining the clustering result according to all the segmentation points includes:
determining a data point in the middle of the segmentation points in the clustering point set as a clustering center;
and mapping the clustering center back to the grid area to obtain the clustering result.
In an optional embodiment of the present invention, for the data points surrounded by the dividing points in the cluster point set, each data point in the cluster point set is a centroid point, that is, each centroid point is an independent cluster, after the binary extraction, each cluster formed by the centroid points is obtained, and then each centroid point is restored to each point in the originally corresponding grid range, which is the final clustering result.
In the embodiment of the invention, a large amount of data is converted into a small number of data points through region division, grid division and particle extraction, so that the algorithm calculation efficiency is greatly improved, and the time cost is saved. And according to the improved OPTICS algorithm and bisection segmentation, the reachable sequence obtained by the improved OPTICS algorithm is iterated, the cluster formed by extracting the centroid points after the segmentation points are obtained is expanded, the density adaptivity is realized, and the dense clusters under different densities can be effectively extracted.
According to the invention, the original service data set is subjected to region division to obtain the original region data set, the original region data set is subjected to grid division, a large amount of data is divided into different grid regions, and the data processing efficiency is improved. Moreover, by calculating the centroid points of the data in the grid, the data can be converted into a relatively small centroid point set, and the data volume is further reduced, so that the clustering processing is performed on the centroid point set, and the clustering efficiency can be improved. Meanwhile, the clustering point set is subjected to category cluster segmentation through a dichotomy, clusters formed by extracting mass center points after segmentation points are obtained are expanded, and the clustering method has density self-adaptability and enables clustering results to be more accurate. Therefore, the data clustering method provided by the invention can solve the problem of low clustering efficiency.
Fig. 2 is a functional block diagram of a data clustering device according to an embodiment of the present invention.
The data clustering device 100 of the present invention can be installed in an electronic device. According to the realized functions, the data clustering device 100 can comprise an area dividing module 101, a grid dividing module 102, a centroid clustering module 103 and a category clustering and splitting module 104. The module of the present invention, which may also be referred to as a unit, refers to a series of computer program segments that can be executed by a processor of an electronic device and that can perform a fixed function, and that are stored in a memory of the electronic device.
In the present embodiment, the functions regarding the respective modules/units are as follows:
the region dividing module 101 is configured to obtain an original service data set, and perform region division on the original service data set to obtain an original region data set;
the mesh division module 102 is configured to perform mesh division on the original region data set, and calculate a centroid point of data in a mesh to obtain a centroid point set;
the centroid clustering module 103 is configured to perform clustering processing on the centroid point set to obtain a clustering point set;
the category cluster segmentation module 104 is configured to perform category cluster segmentation on the cluster point set based on a dichotomy to obtain a clustering result.
In detail, the specific implementation of each module of the data clustering device 100 is as follows:
the method comprises the steps of firstly, obtaining an original service data set, and carrying out region division on the original service data set to obtain an original region data set.
In the embodiment of the present invention, the original service data set may be service data related to a geographic area, for example, in the insurance field, the original service data set may be insurance data of different regions.
Specifically, the performing region division on the original service data set to obtain an original region data set includes:
mapping the service data in the original service data set into service point data in a target area according to the geographical position of the service data in the original service data set;
and summarizing the service point data of all the target areas to obtain the original area data set.
In an optional embodiment of the present invention, taking flooding insurance in the insurance field as an example, according to an accident location of an insurance order, the order is mapped to an insurance data point in a map.
In the embodiment of the invention, when the dense clusters are extracted based on the risk points, the distance measurement between the points uses a geographical distance concept, namely, the clustering based on the spatial distance measurement. When a large amount of point data exists, the point data is directly partitioned in a target area in space, and then algorithm clustering is respectively applied to the point data in each partition in a parallel mode, so that the processing efficiency can be greatly improved. The partitioning mode is usually performed based on administrative divisions, and target areas such as provinces, cities or districts can be selected as partitioning bases by combining with evaluation granularity required by services, so that a large amount of original data are partitioned into different target areas, and the data processing efficiency is improved.
And step two, carrying out grid division on the original region data set, and calculating centroid points of data in grids to obtain a centroid point set.
In the embodiment of the invention, after data partitioning is carried out, in order to further improve the data processing efficiency, the data points in the target area are further subjected to grid division.
In detail, the meshing the original region data set includes:
constructing a graticule by using preset longitude intervals and latitude intervals;
and performing grid division on the service point data in the target area by using the transit network, and summarizing all grid areas containing the service point data.
In the embodiment of the invention, because the data points are not uniformly distributed in the target area, no data point exists in some areas, and the clustering effect is influenced, the data processing efficiency can be improved by screening the data by using the grids.
Specifically, the calculating centroid points of data in the mesh to obtain a centroid point set includes:
selecting any grid area as a target grid, and calculating the centroid points of all service point data in the target grid by using a preset centroid calculation formula;
and assigning the number of the service data points in the target grid to the centroid point as a centroid attribute, and summarizing the assigned centroid points of all grid areas to obtain the centroid point set.
In the embodiment of the invention, a data distribution area is subjected to grid division according to a fixed longitude and latitude interval in a way of dividing a longitude and latitude network in a partition of a target area, after division, a center of mass point is obtained by calculation according to all risk data points in each grid area with risk data points, the total number of the risk data points in each grid area is given to the center of mass point as an attribute value nreports, and after processing, a point set P consisting of a series of center of mass points is obtained, wherein each center of mass point in the point set P is a characteristic point of all risk data points in the corresponding grid area.
In an optional embodiment of the present invention, the calculating the centroid points of all service point data in the target grid by using a preset centroid calculation formula includes:
calculating the centroid points of all service point data in the target grid by using the following centroid calculation formula:
wherein x is the abscissa of the centroid point, y is the ordinate of the centroid point, and xiIs the abscissa, y, of the ith business data pointiIs the ordinate of the ith traffic data point.
In the embodiment of the invention, a large number of service data points can be converted into a relatively small centroid point set through grid division, the data volume is greatly reduced, and the clustering effect is improved.
And step three, clustering the centroid point set to obtain a clustering point set.
In the embodiment of the present invention, since the clustering object is converted into the centroid point set, an improved OPTICS algorithm may be used for clustering, that is: for any center point piContinuously searching the nearest centroid point, and when finding the jth centroid point pjWhen the sum of the nreports attribute value and the central centroid point is greater than MinPts (preset range parameter), then piHas a core distance of piAnd pjDistance between p and piIs consistent with the definition of the original OPTICS algorithm.
In detail, the clustering the centroid point set to obtain a cluster point set includes:
selecting any center of mass point in the center of mass point set as a target center of mass point;
traversing centroid points which are not selected as target centroid points in the centroid point set, and calculating whether the sum of the traversed centroid points and the centroid attributes of the target centroid points is larger than a preset range threshold value or not;
if the sum of the centroid attributes of the centroid point and the target centroid point is not larger than a preset range threshold, reselecting the target centroid point, and returning to the step of traversing the centroid points which are not selected as the target centroid points in the centroid point set;
if the sum of the centroid attributes of the centroid point and the target centroid point is larger than a preset range threshold, determining the target centroid point as a core data point, determining the distance between the centroid point and the target centroid point as a core distance, and calculating the reachable distance of the core data point according to the core distance;
and when all the centroid points are selected as target centroid points, summarizing all the core data points, the core distances of the core data points and the reachable distances of the core data points to obtain the cluster point set.
In the embodiment of the invention, clustering is performed through an improved OPTICS algorithm, and each data point has two attributes, namely: core distance and reach distance.
In an optional embodiment of the present invention, the calculating the reachable distance of the core data point according to the core distance includes:
calculating the reachable distance of the core data point using the following formula:
rd(y，x)＝min{ε：y∈Nε(x) And | Nε(x)|≥MinPts}
Wherein x is a core data point, y is a traversed centroid point, epsilon is a preset field, and N isε(x) MinPts is a predetermined range parameter for the subsamples obtained according to the predetermined field.
In the embodiment of the invention, the clustering point set output by the OPTIC algorithm comprises three ordered arrangements, a core data point sequence, a core distance sequence and an reachable distance sequence.
And fourthly, performing category cluster segmentation on the clustering point set based on a dichotomy to obtain a clustering result.
In the embodiment of the invention, the dichotomy refers to that for a clustering point set, each processing is only divided into two parts, and repeated iteration is carried out for segmentation, so that the data processing efficiency can be improved.
In detail, the performing category cluster segmentation on the clustering point set based on the dichotomy to obtain a clustering result includes:
selecting the data point with the largest reachable distance in the cluster point set as a segmentation point, and segmenting the cluster point set into two category clusters according to the segmentation point;
repeatedly selecting the data point with the largest reachable distance from the two category clusters as a segmentation point to perform iterative segmentation on the two category clusters until a preset iteration condition is met, and stopping segmentation;
and obtaining the clustering result according to all the segmentation points.
In the embodiment of the present invention, for example, after improved OPTICS clustering is performed, an reachable distance sequence S having the same length as the centroid point set P is obtained, at this time, a cluster is extracted, and a segmentation boundary between clusters is selected based on S, which is specifically as follows: 1. finding a maximum value point of the reachable distance sequence S, taking the point as a dividing point, dividing the sequence into a left area S _ left and a right area S _ right, judging whether the number of sample points of the S _ left or the S _ right is less than a preset threshold value or is all null values, and if so, stopping iteration; 2. and (4) repeating the step (1) for the S _ left and the S _ right respectively until iteration is stopped, and finding all the segmentation points.
In detail, the obtaining the clustering result according to all the segmentation points includes:
determining a data point in the middle of the segmentation points in the clustering point set as a clustering center;
and mapping the clustering center back to the grid area to obtain the clustering result.
In an optional embodiment of the present invention, for the data points surrounded by the dividing points in the cluster point set, each data point in the cluster point set is a centroid point, that is, each centroid point is an independent cluster, after the binary extraction, each cluster formed by the centroid points is obtained, and then each centroid point is restored to each point in the originally corresponding grid range, which is the final clustering result.
In the embodiment of the invention, a large amount of data is converted into a small number of data points through region division, grid division and particle extraction, so that the algorithm calculation efficiency is greatly improved, and the time cost is saved. And according to the improved OPTICS algorithm and bisection segmentation, the reachable sequence obtained by the improved OPTICS algorithm is iterated, the cluster formed by extracting the centroid points after the segmentation points are obtained is expanded, the density adaptivity is realized, and the dense clusters under different densities can be effectively extracted.
According to the invention, the original service data set is subjected to region division to obtain the original region data set, the original region data set is subjected to grid division, a large amount of data is divided into different grid regions, and the data processing efficiency is improved. Moreover, by calculating the centroid points of the data in the grid, the data can be converted into a relatively small centroid point set, and the data volume is further reduced, so that the clustering processing is performed on the centroid point set, and the clustering efficiency can be improved. Meanwhile, the clustering point set is subjected to category cluster segmentation through a dichotomy, clusters formed by extracting mass center points after segmentation points are obtained are expanded, and the clustering method has density self-adaptability and enables clustering results to be more accurate. Therefore, the data clustering device provided by the invention can solve the problem of low clustering efficiency.
Fig. 3 is a schematic structural diagram of an electronic device for implementing a data clustering method according to an embodiment of the present invention.
The electronic device may comprise a processor 10, a memory 11, a communication interface 12 and a bus 13, and may further comprise a computer program, such as a data aggregation program, stored in the memory 11 and executable on the processor 10.
The memory 11 includes at least one type of readable storage medium, which includes flash memory, removable hard disk, multimedia card, card-type memory (e.g., SD or DX memory, etc.), magnetic memory, magnetic disk, optical disk, etc. The memory 11 may in some embodiments be an internal storage unit of the electronic device, for example a removable hard disk of the electronic device. The memory 11 may also be an external storage device of the electronic device in other embodiments, such as a plug-in mobile hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), and the like, which are provided on the electronic device. Further, the memory 11 may also include both an internal storage unit and an external storage device of the electronic device. The memory 11 may be used not only to store application software installed in the electronic device and various types of data, such as codes of a data clustering program, but also to temporarily store data that has been output or will be output.
The processor 10 may be composed of an integrated circuit in some embodiments, for example, a single packaged integrated circuit, or may be composed of a plurality of integrated circuits packaged with the same or different functions, including one or more Central Processing Units (CPUs), microprocessors, digital Processing chips, graphics processors, and combinations of various control chips. The processor 10 is a Control Unit (Control Unit) of the electronic device, connects various components of the whole electronic device by using various interfaces and lines, and executes various functions and processes data of the electronic device by running or executing programs or modules (e.g., data clustering programs, etc.) stored in the memory 11 and calling data stored in the memory 11.
The communication interface 12 is used for communication between the electronic device and other devices, and includes a network interface and a user interface. Optionally, the network interface may include a wired interface and/or a wireless interface (e.g., WI-FI interface, bluetooth interface, etc.), which are typically used to establish a communication connection between the electronic device and other electronic devices. The user interface may be a Display (Display), an input unit such as a Keyboard (Keyboard), and optionally a standard wired interface, a wireless interface. Alternatively, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode) touch device, or the like. The display, which may also be referred to as a display screen or display unit, is suitable, among other things, for displaying information processed in the electronic device and for displaying a visualized user interface.
The bus 13 may be a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like. The bus 13 may be divided into an address bus, a data bus, a control bus, etc. The bus 13 is arranged to enable connection communication between the memory 11 and at least one processor 10 or the like.
Fig. 3 shows only an electronic device having components, and those skilled in the art will appreciate that the structure shown in fig. 3 does not constitute a limitation of the electronic device, and may include fewer or more components than those shown, or some components may be combined, or a different arrangement of components.
For example, although not shown, the electronic device may further include a power supply (such as a battery) for supplying power to each component, and preferably, the power supply may be logically connected to the at least one processor 10 through a power management device, so that functions of charge management, discharge management, power consumption management and the like are realized through the power management device. The power supply may also include any component of one or more dc or ac power sources, recharging devices, power failure detection circuitry, power converters or inverters, power status indicators, and the like. The electronic device may further include various sensors, a bluetooth module, a Wi-Fi module, and the like, which are not described herein again.
Further, the electronic device may further include a network interface, and optionally, the network interface may include a wired interface and/or a wireless interface (such as a WI-FI interface, a bluetooth interface, etc.), which are generally used to establish a communication connection between the electronic device and other electronic devices.
Optionally, the electronic device may further comprise a user interface, which may be a Display (Display), an input unit (such as a Keyboard), and optionally a standard wired interface, a wireless interface. Alternatively, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode) touch device, or the like. The display, which may also be referred to as a display screen or display unit, is suitable, among other things, for displaying information processed in the electronic device and for displaying a visualized user interface.
It is to be understood that the described embodiments are for purposes of illustration only and that the scope of the appended claims is not limited to such structures.
The data clustering program stored in the memory 11 of the electronic device is a combination of instructions, and when executed in the processor 10, can realize:
acquiring an original service data set, and performing region division on the original service data set to obtain an original region data set;
carrying out grid division on the original region data set, and calculating centroid points of data in grids to obtain a centroid point set;
clustering the centroid point set to obtain a clustering point set;
and performing category cluster segmentation on the clustering point set based on a dichotomy to obtain a clustering result.
Specifically, the specific implementation method of the instruction by the processor 10 may refer to the description of the relevant steps in the embodiment corresponding to the drawings, which is not described herein again.
Further, the electronic device integrated module/unit, if implemented in the form of a software functional unit and sold or used as a separate product, may be stored in a computer readable storage medium. The computer readable storage medium may be volatile or non-volatile. For example, the computer-readable medium may include: any entity or device capable of carrying said computer program code, recording medium, U-disk, removable hard disk, magnetic disk, optical disk, computer Memory, Read-Only Memory (ROM).
The present invention also provides a computer-readable storage medium, storing a computer program which, when executed by a processor of an electronic device, may implement:
acquiring an original service data set, and performing region division on the original service data set to obtain an original region data set;
carrying out grid division on the original region data set, and calculating centroid points of data in grids to obtain a centroid point set;
clustering the centroid point set to obtain a clustering point set;
and performing category cluster segmentation on the clustering point set based on a dichotomy to obtain a clustering result.
In the embodiments provided in the present invention, it should be understood that the disclosed apparatus, device and method can be implemented in other ways. For example, the above-described apparatus embodiments are merely illustrative, and for example, the division of the modules is only one logical functional division, and other divisions may be realized in practice.
The modules described as separate parts may or may not be physically separate, and parts displayed as modules may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment.
In addition, functional modules in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, or in a form of hardware plus a software functional module.
It will be evident to those skilled in the art that the invention is not limited to the details of the foregoing illustrative embodiments, and that the present invention may be embodied in other specific forms without departing from the spirit or essential attributes thereof.
The present embodiments are therefore to be considered in all respects as illustrative and not restrictive, the scope of the invention being indicated by the appended claims rather than by the foregoing description, and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein. Any reference signs in the claims shall not be construed as limiting the claim concerned.
The embodiment of the application can acquire and process related data based on an artificial intelligence technology. Among them, Artificial Intelligence (AI) is a theory, method, technique and application system that simulates, extends and expands human Intelligence using a digital computer or a machine controlled by a digital computer, senses the environment, acquires knowledge and uses the knowledge to obtain the best result.
The artificial intelligence infrastructure generally includes technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technologies, operation/interaction systems, mechatronics, and the like. The artificial intelligence software technology mainly comprises a computer vision technology, a robot technology, a biological recognition technology, a voice processing technology, a natural language processing technology, machine learning/deep learning and the like.
The block chain is a novel application mode of computer technologies such as distributed data storage, point-to-point transmission, a consensus mechanism, an encryption algorithm and the like. A block chain (Blockchain), which is essentially a decentralized database, is a series of data blocks associated by using a cryptographic method, and each data block contains information of a batch of network transactions, so as to verify the validity (anti-counterfeiting) of the information and generate a next block. The blockchain may include a blockchain underlying platform, a platform product service layer, an application service layer, and the like.
Furthermore, it is obvious that the word "comprising" does not exclude other elements or steps, and the singular does not exclude the plural. A plurality of units or means recited in the system claims may also be implemented by one unit or means in software or hardware. The terms second, etc. are used to denote names, but not any particular order.
Finally, it should be noted that the above embodiments are only for illustrating the technical solutions of the present invention and not for limiting, and although the present invention is described in detail with reference to the preferred embodiments, it should be understood by those skilled in the art that modifications or equivalent substitutions may be made on the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.
Claims (10)
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111525958.9A CN114219023B (en) | 2021-12-14 | 2021-12-14 | Data clustering method, device, electronic device and readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111525958.9A CN114219023B (en) | 2021-12-14 | 2021-12-14 | Data clustering method, device, electronic device and readable storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114219023A true CN114219023A (en) | 2022-03-22 |
| CN114219023B CN114219023B (en) | 2025-04-01 |
Family
ID=80701725
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111525958.9A Active CN114219023B (en) | 2021-12-14 | 2021-12-14 | Data clustering method, device, electronic device and readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114219023B (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114817408A (en) * | 2022-05-10 | 2022-07-29 | 中国平安财产保险股份有限公司 | Scheduling resource identification method and device, electronic equipment and storage medium |
| CN115438138A (en) * | 2022-11-09 | 2022-12-06 | 北京市城市规划设计研究院 | Employment center identification method and device, electronic equipment and storage medium |
| CN115658809A (en) * | 2022-10-17 | 2023-01-31 | 武汉大学 | A data distributed clustering method and device based on local direction centrality |
| CN116756691A (en) * | 2023-06-25 | 2023-09-15 | 国家海洋环境预报中心 | Ocean data assimilation method, ocean data assimilation system, electronic equipment and ocean data assimilation medium |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106682116A (en) * | 2016-12-08 | 2017-05-17 | 重庆邮电大学 | OPTICS point sorting clustering method based on Spark memory computing big data platform |
| CN112347214A (en) * | 2020-11-06 | 2021-02-09 | 平安科技（深圳）有限公司 | Target area dividing method and device, electronic equipment and storage medium |
| CN113011098A (en) * | 2021-03-26 | 2021-06-22 | 云南电网有限责任公司电力科学研究院 | Power transmission corridor area lightning activity rule analysis display model and system |
-
2021
- 2021-12-14 CN CN202111525958.9A patent/CN114219023B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106682116A (en) * | 2016-12-08 | 2017-05-17 | 重庆邮电大学 | OPTICS point sorting clustering method based on Spark memory computing big data platform |
| CN112347214A (en) * | 2020-11-06 | 2021-02-09 | 平安科技（深圳）有限公司 | Target area dividing method and device, electronic equipment and storage medium |
| CN113011098A (en) * | 2021-03-26 | 2021-06-22 | 云南电网有限责任公司电力科学研究院 | Power transmission corridor area lightning activity rule analysis display model and system |
Non-Patent Citations (2)
| Title |
|---|
| CHUNHUA TANG, ET AL.: "An improved OPTICS clustering algorithm for discovering clusters with uneven densities", INTELLIGENT DATA ANALYSIS, vol. 25, no. 6, 29 October 2021 (2021-10-29), pages 1453 - 1471 * |
| 张伟等: "两步聚类方法", 全国第十届企业信息化与工业工程学术年会, 30 November 2006 (2006-11-30), pages 437 - 441 * |
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114817408A (en) * | 2022-05-10 | 2022-07-29 | 中国平安财产保险股份有限公司 | Scheduling resource identification method and device, electronic equipment and storage medium |
| CN114817408B (en) * | 2022-05-10 | 2024-07-02 | 中国平安财产保险股份有限公司 | Scheduling resource identification method and device, electronic equipment and storage medium |
| CN115658809A (en) * | 2022-10-17 | 2023-01-31 | 武汉大学 | A data distributed clustering method and device based on local direction centrality |
| CN115658809B (en) * | 2022-10-17 | 2025-06-24 | 武汉大学 | A data distributed clustering method and device based on local directional centrality |
| CN115438138A (en) * | 2022-11-09 | 2022-12-06 | 北京市城市规划设计研究院 | Employment center identification method and device, electronic equipment and storage medium |
| CN115438138B (en) * | 2022-11-09 | 2023-04-07 | 北京市城市规划设计研究院 | Employment center identification method and device, electronic equipment and storage medium |
| CN116756691A (en) * | 2023-06-25 | 2023-09-15 | 国家海洋环境预报中心 | Ocean data assimilation method, ocean data assimilation system, electronic equipment and ocean data assimilation medium |
| CN116756691B (en) * | 2023-06-25 | 2024-01-30 | 国家海洋环境预报中心 | Ocean data assimilation method, ocean data assimilation system, electronic equipment and ocean data assimilation medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114219023B (en) | 2025-04-01 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN114219023B (en) | Data clustering method, device, electronic device and readable storage medium | |
| Long et al. | Mapping block-level urban areas for all Chinese cities | |
| CN106844781B (en) | Data processing method and device | |
| CN113516417A (en) | Service evaluation method and device based on intelligent modeling, electronic equipment and medium | |
| CN113946690A (en) | Potential customer mining method and device, electronic equipment and storage medium | |
| CN112990374A (en) | Image classification method, device, electronic equipment and medium | |
| CN111768096A (en) | Rating method and device based on algorithm model, electronic equipment and storage medium | |
| CN111783982A (en) | Attack sample acquisition method, device, equipment and medium | |
| CN113961473A (en) | Data testing method and device, electronic equipment and computer readable storage medium | |
| CN113327136A (en) | Attribution analysis method and device, electronic equipment and storage medium | |
| CN114913371A (en) | Multitask learning model training method and device, electronic equipment and storage medium | |
| CN111950621A (en) | Target data detection method, device, equipment and medium based on artificial intelligence | |
| CN117155771B (en) | Equipment cluster fault tracing method and device based on industrial Internet of things | |
| CN113434542B (en) | Data relationship identification method and device, electronic equipment and storage medium | |
| CN111930897A (en) | Patent retrieval method, device, electronic equipment and computer-readable storage medium | |
| CN114168795A (en) | Building three-dimensional model mapping and storing method and device, electronic equipment and medium | |
| CN114201482A (en) | Demographic distribution statistics method, device, electronic device and readable storage medium | |
| CN114398194A (en) | Data collection method and device, electronic equipment and readable storage medium | |
| CN113868529A (en) | Knowledge recommendation method and device, electronic equipment and readable storage medium | |
| CN111950707A (en) | Behavior prediction method, apparatus, device and medium based on behavior co-occurrence network | |
| CN114840631A (en) | Spatial text query method and device, electronic equipment and storage medium | |
| CN112529086B (en) | Stop line generation method, electronic device, and storage medium | |
| CN114238375A (en) | Index query method and device, electronic equipment and storage medium | |
| CN115757987B (en) | Method, device, equipment and medium for determining companion object based on track analysis | |
| CN116664416A (en) | Radar point cloud data processing method, device, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |