CN108197323B

CN108197323B - Map data processing method applied to distributed system

Info

Publication number: CN108197323B
Application number: CN201810110349.9A
Authority: CN
Inventors: 丁武轩
Original assignee: Shenzhen Etop Information Co ltd
Current assignee: SHENZHEN ETOP INFORMATION Co.,Ltd.
Priority date: 2018-02-05
Filing date: 2018-02-05
Publication date: 2021-03-30
Anticipated expiration: 2038-02-05
Also published as: CN108197323A

Abstract

The invention provides a map data processing method applied to a distributed system, wherein the distributed system is composed of a plurality of distributed clusters, each distributed cluster comprises a branch management server and a plurality of data servers, the branch management server is used for managing the data servers, and the map data processing method comprises the following steps: uploading map data in response to an instruction to upload the map data to the distributed system; and in a preset time, the distributed system automatically performs two rounds of backup on the map data. The invention provides a complete distributed system and a complete method for uploading, backing up and processing map data in the distributed system, thereby solving the technical problems of massive map data storage management and data processing.

Description

Map data processing method applied to distributed system

Technical Field

The invention relates to the field of data processing, in particular to a map data processing method applied to a distributed system.

Background

The data volume of the map data is huge, high requirements are placed on a server and a terminal in storage, processing and rendering, the timeliness of data switching and map rendering is very important for a user, and the difference of the second level is enough to seriously reduce the user experience;

in order to increase the storage, management and processing capacity of map data, a distributed network is used as a feasible way, but the technical requirement of the distributed network is high, and the algorithm requirement on data consistency and backup is also high; in addition, even though the problem of data storage management is solved based on a distributed network, the huge data volume of map data also brings heavy burden to terminal rendering.

Disclosure of Invention

In order to solve the technical problem, the invention provides a map data processing method applied to a distributed system.

The invention is realized by the following technical scheme:

the map data processing method is applied to a distributed system, the distributed system is composed of a plurality of distributed clusters, each distributed cluster comprises a branch management server and a plurality of data servers, the branch management server is used for managing the data servers, and the map data processing method comprises the following steps:

uploading map data in response to an instruction to upload the map data to the distributed system;

and in a preset time, the distributed system automatically performs two rounds of backup on the map data.

Further, in response to an instruction for uploading map data to a distributed system, performing primary hashing on the map data to obtain a target storage node for storing the map data;

judging whether the target storage node is a current available node or not;

if yes, allowing uploading and receiving the map data by the target storage node;

and if the target storage node is the current unavailable node, acquiring the abstract of the file where the map data is located, and performing secondary hashing according to the abstract to obtain a hash value.

Further, carrying out one-time hashing according to the creation time to obtain a hash value; and carrying out secondary hashing according to the digest to obtain a hash value.

Further, in the data backup method, a timer is generated by the distributed system from the time of successful uploading, so that the backup of the map data is completed within a preset time.

Further, the setting of the preset time refers to a result of an analysis of a behavior of the distributed system in response to the user accessing the data.

Further, the preset time for the vector map data should be less than 2 days, and the preset time for the image map data should be less than 5 days.

The invention has the beneficial effects that:

the map data processing method applied to the distributed system has the following beneficial effects:

(1) a complete distributed system and a complete method for uploading, backing up and processing map data in the distributed system are provided, so that the technical problems of massive map data storage management and data processing are solved;

(2) in order to accelerate the rendering speed of the map data by the mobile terminal, a method for generating the simplified file by thinning the map data on the premise of saving the topological relation of the map data is provided, so that the terminal can render the simplified file during summary browsing and render the original file during fine browsing, thereby balancing the rendering speed and the rendering effect which a user wants to see.

Drawings

Fig. 1 is a flowchart of a map data processing method applied to a distributed system according to an embodiment of the present invention;

FIG. 2 is a block diagram of a distributed system provided by an embodiment of the invention;

FIG. 3 is a flowchart of a backup method provided by an embodiment of the invention;

FIG. 4 is a flowchart of a method for processing map data according to an embodiment of the present invention;

FIG. 5 is a flowchart of a comprehensive processing method for small patches according to an embodiment of the present invention;

FIG. 6 is a flow chart of a simplified processing method provided by an embodiment of the present invention;

FIG. 7 is a flowchart of a polyline compression algorithm provided by an embodiment of the invention.

Detailed Description

In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention will be described in further detail with reference to the accompanying drawings.

The embodiment of the invention provides a map data processing method applied to a distributed system, and as shown in fig. 1, the method comprises the following steps:

in response to an instruction of uploading map data to a distributed system, carrying out primary hashing on the map data to obtain a target storage node for storing the map data;

judging whether the target storage node is a current available node or not;

and if so, allowing the map data to be uploaded and received by the target storage node.

The distributed system has a structure as shown in fig. 2, and is formed by a plurality of distributed clusters, each distributed cluster including a sub-management server and a plurality of data servers, and the sub-management server is used for managing the data servers. Namely, the sub-management server forms a name node of the distributed cluster, the data server forms a data node of the distributed cluster, one name node and the data node governed by the name node form the distributed cluster, and the sum of all the distributed clusters forms a distributed system and is governed by a management server in the distributed system. In the distributed system, each distributed cluster is provided with a corresponding cluster identifier, the identifier of each name node corresponds to the cluster identifier, and each data node identifier is composed of a name node identifier and a distinguishing code.

When a user issues an instruction for uploading map data to the distributed system, the creation time and the file size of a file where the map data are located are obtained, a hash value is obtained through one-time hashing according to the creation time, and a status request is sent to a data node (target storage node) corresponding to the hash value, so that the data node can return status data, wherein the status data comprises whether the data are available and the remaining storage space.

And if the data node is in an available state and the file size is smaller than the residual storage space of the data node, the target storage node is the current available node, and uploading is allowed and the target storage node receives the map data.

And if the state data is unavailable or the file size is not smaller than the residual storage space of the data node, the target storage node is the current unavailable node. The method comprises the steps of obtaining an abstract of a file where the map data are located, conducting secondary hashing according to the abstract to obtain a hash value, and sending a state request to a data node (target storage node) corresponding to the hash value so that the data node can return state data, wherein the state data comprise whether the data node is available and the residual storage space. And if the data node is in an available state and the file size is smaller than the residual storage space of the data node, the target storage node is the current available node, and uploading is allowed and the target storage node receives the map data.

Further, in order to avoid data loss, an embodiment of the present invention provides a data backup method.

In the data backup method, a timer is generated by the distributed system from the time of successful uploading, so that the backup of the map data is completed within a preset time. The setting of the preset time refers to the result of an analysis of the behavior of the distributed system in response to user access data.

For example, based on analysis of the access log maintained in the distributed system, it is known that 80% of the vector map data will be accessed within two days of uploading, and 85% of the image map data will be accessed after 5 days of uploading, so the preset time for the vector map data should be less than 2 days, and the preset time for the image map data should be less than 5 days. Obviously, with the use of the distributed system, the result of the analysis based on the access log is updated, and accordingly, the preset time is changed.

As shown in fig. 3, in the first round of backup process, the uploading time of the map data is obtained, the uploading time is hashed in the target cluster space in the distributed system, a target backup node corresponding to the hash value is obtained, whether the target backup node is a current available node is determined (the determination method is as described above), and if yes, a first copy of the map data is generated at the target backup node. Specifically, the target cluster space is a sum of other distributed clusters except for a distributed cluster where a data node where the map data is planed is located in the distributed system.

In the second round of backup process, the byte number of the map data is obtained, the byte number is hashed in the target data node space in the distribution system to obtain a target backup node corresponding to the hash value, whether the target backup node is a current available node is judged (the judgment method is as described above), and if yes, a second copy of the map data is generated at the target backup node. Specifically, the target data node space is a sum of data nodes obtained by removing the data nodes where the first copy is located in the distributed cluster where the node where the first copy is located.

On the basis that a distributed system stores and backs up acquired map data, the embodiment of the invention provides a method for processing the map data, wherein the map data is vector data, and the vector data comprises spot position data and attribute data.

The method is shown in fig. 4 and comprises the following steps:

carrying out comprehensive treatment on small image spots;

simplifying the vector data after comprehensive processing for multiple times respectively; the simplification scales in each simplification process are different, and the simplification scales are identified by an execution threshold in the simplification process;

and generating a simplified file aiming at each simplified result.

Specifically, the small-pattern-spot comprehensive processing is shown in fig. 5, and includes:

acquiring all minimum outsourcing rectangles of the image spots;

acquiring all target pattern spots of which the minimum outsourcing rectangle is smaller than a preset area value;

for each target spot, performing the following operations:

(1) judging whether the target pattern spot has a ground class code, if not, deleting the target pattern spot;

(2) if the target pattern spot exists, the land class code of the adjacent pattern spot of the target pattern spot is obtained, and if the adjacent pattern spot exists, the target pattern spot is merged to the adjacent pattern spot, wherein the land class code of the adjacent pattern spot of the target pattern spot is the same as or similar to the land class code of the target pattern spot; otherwise, deleting the target image spot.

To simplify vector data, an embodiment of the present invention provides a simplified processing method as shown in fig. 6, including:

and S1, storing the vector data into a preset first data structure to generate original data.

And S2, compressing the original data according to a preset algorithm to obtain compressed target data.

And S3, storing the target data into a preset second data structure to obtain a simplified file.

Further, before S1, there may be a step of filtering each patch in the vector data to filter out repeated position data.

Next, step S1 is described in detail:

in the embodiment of the present invention, vector data is described using a first data structure, specifically, the first data structure is a quadtree structure, and each node identifies a blob. The topology of the quadtree reflects the spatial position relationship of the pattern spots.

Next, step S2 is described in detail:

in a map displayed by using vector data as a data source, planar patches are distributed without overlapping and without gaps, and the patches are directly subjected to lossy compression, so that the compressed patches are cracked due to the loss of some position data, and the effect of the non-overlapping and seamless distribution cannot be maintained.

In a node, a vector container is used for recording a pattern spot, each element in the vector container is a linked list, each linked list node in the linked list represents a section of broken line in the pattern spot, in order to avoid pattern spot cracks, data preprocessing needs to acquire pattern spot topology, the pattern spot topology comprises a common edge of the pattern spot and an adjacent pattern spot, and in order to acquire the common edge, each linked list node records the following contents:

the pattern spot is marked;

the first starting identifier is used for recording the starting of the pattern spot broken line;

the first end mark is used for recording the end of the pattern spot broken line;

matching the pattern spot identification, wherein the matching pattern spot is a pattern spot which is extracted by a common edge with the pattern spot for solving topology; the obvious matching icon is the adjacent image spot of the image spot;

a second start identifier for recording the start of the adjacent spot polyline;

the second ending mark is used for recording the ending of the adjacent pattern spot broken lines;

and the public edge identifier is used for identifying whether the broken line recorded by the linked list node is a public edge of the graph spot and the adjacent graph spots.

In the topology solving process, when a certain linked list node is processed, whether a section of broken line of the graph spot and a section of broken line of the matched graph spot are overlapped or not is judged, and if all the positions of the broken line and the section of broken line are found to be overlapped, the public edge identification is marked. If the partial superposition is found, splitting the linked list nodes according to the superposition part, specifically:

if the overlapped part is positioned at the beginning or the end of the pattern spot broken line, the linked list is split into two target linked lists, one target linked list records a common edge, and the other target linked list records a non-common edge.

If the overlapped part is positioned in the middle of the pattern spot broken line, the linked list is divided into three target linked lists, the middle target linked list records a common edge, and other target linked lists record non-common edges.

After extracting the common edges of all the image spots, the topology of the original data can be obtained, compression is carried out according to the topology, namely compression algorithms are respectively carried out on the common edges and the non-common edges, the results after the compression are respectively stored into preset second data, and a simplified file is generated.

The same or different compression algorithms may be used for the compression of the common edge and the non-common edge, and an embodiment of the present invention provides a feasible polyline compression algorithm, as shown in fig. 7, which includes:

(1) a straight line AB is connected between the head point A and the tail point B of the broken line, and the straight line is a chord of the curve;

(2) obtaining a point C with the maximum distance from the straight line segment on the broken line, and calculating the distance d between the point C and the AB;

(3) comparing the distance with the size of an execution threshold, and if the distance is smaller than the execution threshold, deleting C;

(4) and if the distance is greater than the execution threshold, dividing the broken line into two segments of AC and BC by using C, and respectively carrying out processing of (1) - (3) on the two segments of broken line.

It is necessary to mainly change the execution threshold to perform multiple times of compression on the premise of obtaining the original data topology, so as to obtain compressed data with different scales. And storing each compressed data according to a preset second data structure to obtain simplified files with different scales.

Obviously, the simplified file is useful in a map display scene, and when a user needs to observe summary data, the simplified file can be used for displaying a map, so that the map can be quickly rendered, and the user can quickly acquire the general map; when the user needs to observe the detail data, the original data can be used, so that the map detail can be displayed without loss. Obviously, during the zooming-in operation or zooming-out operation of the user, the simplified files or the original data with different scales can be loaded according to the zooming scale.

In the description provided herein, numerous specific details are set forth. It is understood, however, that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures and techniques have not been shown in detail in order not to obscure an understanding of this description.

Similarly, it should be appreciated that in the foregoing description of exemplary embodiments of the invention, various features of the invention are sometimes grouped together in a single embodiment, figure, or description thereof for the purpose of streamlining the disclosure and aiding in the understanding of one or more of the various inventive aspects. However, the disclosed method should not be interpreted as reflecting an intention that: that the invention as claimed requires more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive aspects lie in less than all features of a single foregoing disclosed embodiment. Thus, the claims following the detailed description are hereby expressly incorporated into this detailed description, with each claim standing on its own as a separate embodiment of this invention.

Those skilled in the art will appreciate that the modules in the device in an embodiment may be adaptively changed and disposed in one or more devices different from the embodiment. The modules or units or components of the embodiments may be combined into one module or unit or component, and furthermore they may be divided into a plurality of sub-modules or sub-units or sub-components. All of the features disclosed in this specification (including any accompanying claims, abstract and drawings), and all of the processes or elements of any method or apparatus so disclosed, may be combined in any combination, except combinations where at least some of such features and/or processes or elements are mutually exclusive. Each feature disclosed in this specification (including any accompanying claims, abstract and drawings) may be replaced by alternative features serving the same, equivalent or similar purpose, unless expressly stated otherwise.

Furthermore, those skilled in the art will appreciate that although embodiments described herein include some features included in other embodiments, not other features, combinations of features of different embodiments are meant to be within the scope of the invention and form different embodiments. For example, in the claims of the present invention, any of the claimed embodiments may be used in any combination.

The present invention may also be embodied as apparatus or system programs (e.g., computer programs and computer program products) for performing a portion or all of the methods described herein. Such programs implementing the present invention may be stored on computer-readable media or may be in the form of one or more signals. Such a signal may be downloaded from an internet website, provided on a carrier signal, or provided in any other form.

It should be noted that the above-mentioned embodiments illustrate rather than limit the invention, and that those skilled in the art will be able to design alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The word "comprising" does not exclude the presence of elements or steps or the like not listed in a claim. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The invention may be implemented by means of hardware comprising several distinct elements, and by means of a suitably programmed computer. In the unit claims enumerating several systems, several of these systems may be embodied by one and the same item of hardware. The usage of the words first, second and third, etcetera do not indicate any ordering and these words may be interpreted as names.

Claims

1. The map data processing method applied to the distributed system is characterized in that the distributed system is composed of a plurality of distributed clusters, each distributed cluster comprises a branch management server and a plurality of data servers, the branch management server is used for managing the data servers, and the map data processing method comprises the following steps:

in response to an instruction of uploading map data to a distributed system, carrying out primary hashing on the map data to obtain a target storage node for storing the map data; judging whether the target storage node is a current available node or not; if yes, allowing uploading and receiving the map data by the target storage node; within a preset time, the distributed system automatically performs two rounds of backup for map data; the sub-management servers form name nodes of the distributed cluster, the data servers form data nodes of the distributed cluster, one name node and the data nodes governed by the name node form the distributed cluster, and the sum of all the distributed clusters forms a distributed system and is governed by a management server in the distributed system; in the distributed system, each distributed cluster is provided with a corresponding cluster identifier, the identifier of each name node corresponds to the cluster identifier, and each data node identifier consists of a name node identifier and a distinguishing code;

when a user issues an instruction for uploading map data to the distributed system, acquiring creation time and file size of a file where the map data are located, performing primary hashing according to the creation time to obtain a hash value, and sending a status request to a target storage node corresponding to the hash value so as to enable the data node to return status data, wherein the status data comprises whether the data are available and a residual storage space; if the data node is in an available state and the file size is smaller than the remaining storage space of the data node, the target storage node is a current available node, uploading is allowed, and the target storage node receives the map data;

if the state data is unavailable or the file size is not smaller than the residual storage space of the data node, the target storage node is a current unavailable node; acquiring the abstract of the file where the map data is located, performing secondary hashing according to the abstract to obtain a hashed value, and sending a state request to a target storage node corresponding to the hashed value so as to facilitate the data node to return state data, wherein the state data comprises whether the data node is available and the residual storage space; if the data node is in an available state and the file size is smaller than the remaining storage space of the data node, the target storage node is a current available node, uploading is allowed, and the target storage node receives the map data;

within a preset time, the distributed system automatically performs two rounds of backup for map data;

in the first round of backup process, obtaining uploading time of map data, hashing the uploading time in a target cluster space in the distributed system to obtain a target backup node corresponding to a hash value, judging whether the target backup node is a current available node or not, and if so, generating a first copy of the map data at the target backup node; specifically, the target cluster space is the sum of other distributed clusters except the distributed cluster where the data node where the map data is planed is located in the distributed system;

in the second round of backup process, acquiring byte number of map data, hashing the byte number in a target data node space in the distributed system to obtain a target backup node corresponding to a hash value, judging whether the target backup node is a current available node or not, and if so, generating a second copy of the map data at the target backup node; the target data node space is the sum of data nodes obtained after the data nodes where the first copy is located are planed in the distributed cluster where the node where the first copy is located.

2. The map data processing method applied to the distributed system according to claim 1, wherein: in the data backup method including the first round of backup and the second round of backup, the distributed system generates a timer since the uploading is successful, so that the backup for the map data is completed within a preset time.

3. The map data processing method applied to the distributed system according to claim 2, wherein: the setting of the preset time refers to the result of an analysis of the behavior of the distributed system in response to user access data.

4. The map data processing method applied to the distributed system according to claim 3, wherein: the preset time for the vector map data should be less than 2 days, and the preset time for the image map data should be less than 5 days.