CN111274228B - Policy data migration storage method, system, equipment and readable storage medium - Google Patents
Policy data migration storage method, system, equipment and readable storage medium Download PDFInfo
- Publication number
- CN111274228B CN111274228B CN202010108664.5A CN202010108664A CN111274228B CN 111274228 B CN111274228 B CN 111274228B CN 202010108664 A CN202010108664 A CN 202010108664A CN 111274228 B CN111274228 B CN 111274228B
- Authority
- CN
- China
- Prior art keywords
- node
- index information
- nodes
- policy data
- capacity expansion
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/214—Database migration support
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q40/00—Finance; Insurance; Tax strategies; Processing of corporate or income taxes
- G06Q40/08—Insurance
Abstract
The invention provides a policy data migration storage method, a system, equipment and a readable storage medium, wherein the threshold value of the number of nodes is modulo according to the policy identification value, and the grouping sequence number of the node position stored before the initial node is not expanded is determined for each policy data, so that initial index information is established, the index information is adjusted, data migration storage is carried out according to the adjusted index information, the capacity expansion can be carried out on the basis of the existing distributed storage system, the number of newly-increased nodes and the data migration quantity required by each capacity expansion are reduced, the dependence on the direct hash modulo storage of the policy identification value in the existing capacity expansion scheme is relieved, and the flexibility and the reliability of the policy data migration storage are improved.
Description
Technical Field
The present invention relates to data storage technologies, and in particular, to a policy data migration storage method, system, device, and readable storage medium.
Background
After underwriting, the insurance company signs an insurance contract with the applicant, and the insurance policy is stored in a database of a background system in an electronic mode. With the development of internet insurance, insurance can be purchased more conveniently and in a more manner by an applicant. And the fragmentation scenes of a large number of short risks and extremely short risks are more and more, so that a large number of policy data storage requirements are brought.
The existing policy data storage management mode adopts a single machine and centralized architecture mode. In general, a distributed hash modular approach is adopted to store policy data, and the node serial number of the policy data storage can be obtained by tracing the result of the modular approach of the identification value of each policy data to the total number of storage nodes. When the capacity of the nodes needs to be increased, the node quantity is increased by adopting a node fission mode, half of policy data is synchronously migrated to the newly added nodes, and the tracing mode of the policy storage nodes is unchanged.
According to the prior art, a plurality of nodes are required to be added for each capacity expansion, half of policy data are required to be migrated, convenience is low, and data migration risk is high.
Disclosure of Invention
The embodiment of the invention provides a policy data migration storage method, a system, equipment and a readable storage medium, which remove the dependence on direct hash modular storage of a policy identification value in the existing capacity expansion scheme and improve the flexibility and reliability of policy data migration storage.
According to a first aspect of the present invention, there is provided a policy data migration storage method, including:
acquiring a policy identification value corresponding to policy data stored by each node before capacity expansion and a preset node quantity threshold, wherein the node serial number of the node stored by each policy data before primary capacity expansion is a value obtained by modulo the policy identification value of the policy data on the number of the nodes before capacity expansion;
respectively modulo the number threshold of the nodes by the policy identification value to obtain the grouping sequence numbers corresponding to the policy data, wherein the number threshold of the nodes is 2 of the number of the nodes before capacity expansion N Multiple, N is an integer greater than 0;
when the capacity expansion of a newly added node is determined, acquiring expanded first index information, wherein the first index information comprises the node serial numbers of the nodes after the capacity expansion and the group serial numbers corresponding to the node serial numbers, and the nodes corresponding to the group serial numbers are used for storing policy data corresponding to the group serial numbers;
and responding to the adjustment of the corresponding grouping sequence numbers of the sequence numbers of all nodes in the first index information, and performing migration storage on the policy data in all nodes according to the adjusted first index information.
According to a second aspect of the present invention, there is provided a policy data migration storage system comprising:
the system comprises an acquisition module, a storage module and a storage module, wherein the acquisition module is used for acquiring a policy identification value corresponding to policy data stored by each node before capacity expansion and a preset node quantity threshold value, wherein the node serial number of the node stored by each policy data before primary capacity expansion is a value obtained by modulo the policy identification value of the policy data on the node quantity before capacity expansion;
the grouping module is used for respectively modulo the number threshold of the nodes by the policy identification value to obtain the grouping sequence number corresponding to each policy data, wherein the number threshold of the nodes is 2 of the number of the nodes before capacity expansion N Multiple, N is an integer greater than 0;
the capacity expansion module is used for acquiring expanded first index information when the capacity expansion of the newly added node is determined, wherein the first index information comprises the node serial numbers of the nodes after the capacity expansion and the group serial numbers corresponding to the node serial numbers, and the nodes corresponding to the group serial numbers are used for storing policy data corresponding to the group serial numbers;
and the processing module is used for responding to the adjustment of the corresponding grouping sequence numbers of the nodes in the first index information, and performing migration storage on the policy data in the nodes according to the adjusted first index information.
According to a third aspect of the present invention, there is provided an electronic device comprising: a memory, a processor and a computer program stored in the memory, the processor running the computer program to perform the policy data migration storage method of the first aspect and the various possible designs of the first aspect.
According to a fourth aspect of the present invention there is provided a computer readable storage medium having stored therein a computer program for carrying out the policy data migration storage method of the first aspect and the various possible designs of the first aspect of the present invention when executed by a processor.
The embodiment of the invention provides a policy data migration storage method, a system, equipment and a readable storage medium, which are characterized in that a policy identification value corresponding to policy data stored by each node before capacity expansion and a preset node number threshold value are obtained, wherein the node serial number of the node stored by each policy data before primary capacity expansion is a value obtained by taking a module of the policy identification value of the policy data to the number of the nodes before capacity expansion; respectively modulo the number threshold of the nodes by the policy identification value to obtain the grouping sequence numbers corresponding to the policy data, wherein the number threshold of the nodes is 2 of the number of the nodes before capacity expansion N Multiple, N is an integer greater than 0; when the capacity expansion of a newly added node is determined, acquiring expanded first index information, wherein the first index information comprises the node serial numbers of the nodes after the capacity expansion and the group serial numbers corresponding to the node serial numbers, and the nodes corresponding to the group serial numbers are used for storing policy data corresponding to the group serial numbers; and in response to the adjustment of the corresponding grouping sequence numbers of the node sequence numbers in the first index information, migration storage is carried out on the policy data in each node according to the adjusted first index information, so that the threshold value of the node quantity is modulo according to the policy identification value, the grouping sequence numbers which can trace back to the node positions stored before the initial node is not expanded are determined for each policy data, the initial index information is built, the index information is adjusted, data migration storage is carried out according to the adjusted index information, capacity expansion can be carried out on the basis of the existing distributed storage system, the number of newly added nodes and the data migration quantity required by capacity expansion at each time are reduced, the dependence on direct hash modulo storage of the policy identification value in the existing capacity expansion scheme is relieved, and the flexibility and reliability of policy data migration storage are improved.
Drawings
FIG. 1 is a schematic diagram of a prior art data migration method;
FIG. 2 is a schematic flow chart of a policy data migration and storage method according to an embodiment of the present invention;
FIG. 3 is an example of initial index information establishment provided by an embodiment of the present invention;
fig. 4 is a block sequence number adjustment example provided by an embodiment of the present invention;
FIG. 5 is a block diagram of a policy data migration storage system provided by the present invention;
fig. 6 is a schematic hardware structure of an electronic device according to an embodiment of the present invention.
Detailed Description
For the purpose of making the objects, technical solutions and advantages of the embodiments of the present invention more apparent, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention, and it is apparent that the described embodiments are only some embodiments of the present invention, not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
The terms first, second and the like in the description and in the claims and in the above-described figures, are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged where appropriate such that the embodiments of the invention described herein may be implemented in sequences other than those illustrated or otherwise described herein.
It should be understood that, in various embodiments of the present invention, the sequence number of each process does not mean that the execution sequence of each process should be determined by its functions and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.
It should be understood that in the present invention, "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements that are expressly listed or inherent to such process, method, article, or apparatus.
It should be understood that in the present invention, "plurality" means two or more. "and/or" is merely an association relationship describing an association object, and means that three relationships may exist, for example, and/or B may mean: a exists alone, A and B exist together, and B exists alone. The character "/" generally indicates that the context-dependent object is an "or" relationship. "comprising A, B and C", "comprising A, B, C" means that all three of A, B, C comprise, "comprising A, B or C" means that one of the three comprises A, B, C, and "comprising A, B and/or C" means that any 1 or any 2 or 3 of the three comprises A, B, C.
It should be understood that in the present invention, "B corresponding to a", "a corresponding to B", or "B corresponding to a" means that B is associated with a, from which B can be determined. Determining B from a does not mean determining B from a alone, but may also determine B from a and/or other information. The matching of A and B is that the similarity of A and B is larger than or equal to a preset threshold value.
As used herein, "if" may be interpreted as "at … …" or "at … …" or "in response to a determination" or "in response to detection" depending on the context.
The technical scheme of the invention is described in detail below by specific examples. The following embodiments may be combined with each other, and some embodiments may not be repeated for the same or similar concepts or processes.
Referring to fig. 1, a schematic diagram of a conventional data migration method is shown. The policy identification value of the policy data may be a unique identification of the policy data, for example, a numerical value of the policy ID. With the increase of the quantity of the policy, the prior art builds a distributed storage system formed by a plurality of nodes, and stores and inquires the policy by taking the node serial numbers and the policy IDs as indexes. In the prior art, data storage is generally performed by using 2K-th power nodes, for example, 4 storage nodes are used, and node serial numbers are node 0, node 1, node 2 and node 3. The node serial numbers stored in each policy may be obtained by modulo the number of nodes (e.g., 4 nodes before capacity expansion as shown in fig. 1) with the policy ID. Taking a policy ID18 as shown in fig. 1 as an example, 18 modulo 4 gives 2, i.e., the policy ID18 is stored in node 2. The nodes stored in each policy can be traced back through modulo arithmetic. When the capacity expansion is needed, the capacity expansion mode of the existing node architecture is node fission, namely the number of nodes is changed from the power of K of 2 to the power of K+1 of 2. For example, 8 nodes changed from the initial 4 nodes to capacity expansion are shown in fig. 1, wherein, node 0 is split into node 0 and node 4, node 1 is split into node 1 and node 5, node 2 is split into node 6, node 3 is split into node 3 and node 7, half of the policy data is reserved by the original node, and the other half of the policy data is migrated to the newly added data node, as shown in fig. 1. After the node is expanded, the requirement that the policy ID takes the modulo and traces back the stored node serial number is still met.
Therefore, in the existing scheme, not only a larger data volume needs to be migrated during capacity expansion, but also a new node with the same number as the existing node needs to be added, so that higher machine cost is required, and the larger data migration volume has higher security risk. In order to solve the problems in the prior art, the invention provides a policy data migration storage method, which expands capacity based on the existing distributed storage system, reduces the number of newly added nodes and data migration volume required by each expansion, reduces system risk, removes the dependence on direct hash modular storage of policy identification values in the existing expansion scheme, and improves the flexibility and reliability of policy data migration storage.
Referring to fig. 2, a flowchart of a policy data migration storage method provided by an embodiment of the present invention, an execution body of the method shown in fig. 2 may be a software and/or hardware device, for example, a distributed data management system, or a background management server for controlling data reading and writing of each node, which is not limited herein. The method shown in fig. 2 includes steps S101 to S104, which are specifically as follows:
s101, acquiring a policy identification value corresponding to policy data stored by each node before capacity expansion and a preset node quantity threshold, wherein the node serial number of the node stored by each policy data before primary capacity expansion is a value obtained by modulo the policy identification value of the policy data on the node quantity before capacity expansion.
A node is understood to be a node storing data and may in particular be a device for storing data, such as a memory, a data server, etc. When the initial construction node stores policy data, the number of the initial nodes is the power of 2 and K is an integer greater than 0. And the initial number of nodes to the power of 2 is less than the threshold number of nodes. Taking the initial node number shown in fig. 1 as 4 as an example, the node serial number of the node stored in each policy data before the initial capacity expansion is a value obtained by taking the modulo of the policy identification value to 4. As shown in the figure, if the policy ID12 modulo 4 gets 0, the policy data corresponding to the policy ID12 is stored in the node 0; taking the modulus of the policy ID13 to 4 to obtain 1, and storing the policy data corresponding to the policy ID13 in the node 1; taking the modulus of the policy ID14 to the 4 to obtain 2, and storing the policy data corresponding to the policy ID14 in the node 2; and taking the modulus of the policy ID15 to 4 to obtain 3, and storing the policy data corresponding to the policy ID15 in the node 3.
The preset threshold value of the number of the nodes can be estimated to be the maximum capacity according to the limit order of the possible quantity of the policy and the storage capacity of each node, for example, the threshold value of the number of the nodes can be 2 8 I.e. the nodes extend up to 512. However, the present embodiment is not limited thereto, and the threshold value of the number of nodes may be 1024, 2048 or the like.
In the existing distributed storage system established based on hash fission, an index relation between each node and each policy identification value under the initial node quantity is required to be established before capacity expansion, the relation between the policy identification values and the packets is established firstly, and then the relation between the packets and the nodes is defined, so that two-stage indexing of policy data is realized, and dependence on direct hash modular storage of the policy identification values in the existing capacity expansion scheme is relieved.
S102, respectively modulo the number threshold of the nodes by the policy identification value to obtain the group serial number corresponding to each policy dataWherein the threshold of the node number is 2 of the node number before capacity expansion N Multiple, N is an integer greater than 0.
Referring to fig. 3, an initial index information establishment example is provided in an embodiment of the present invention. In the example shown in fig. 3, the threshold of the number of nodes is 512, and 512 packets are obtained after the policy identification value of each stored policy data is modulo 512, that is, packet numbers 0 to 511 in the initial index information shown in fig. 3. If the policy identification value is 1, 2, …, 512, the 512 is modulo to obtain the packet sequence numbers 1, 2, …, 511, 0 in sequence. Then, the policy identification values 513, 514, … 1024 modulo 512 again obtain packet numbers 1, 2, …, 511, 0. And so on, determining the corresponding packet sequence number for each policy identification value. As shown in fig. 3, the value obtained by modulo the packet number 4 (the number of the initial nodes in fig. 3) is the node number corresponding to each packet stored in the number of the initial nodes when the capacity is not expanded. In the prior art, the node serial numbers of the policy data storage nodes are determined by taking the modulus of the number of the nodes by the policy identification value, the grouping serial numbers are obtained by taking the modulus of the threshold value of the number of the nodes by the policy identification value, and the result of taking the modulus of the initial number of the grouping serial numbers is consistent with the node serial numbers obtained in the prior art, so that backtracking of the policy data storage positions can be realized. Whether the capacity is expanded for several times and the number of the nodes is a fixed value, the threshold value of the number of the nodes is respectively modulo the threshold value of the number of the nodes through the policy identification value, the grouping of the policy identification value is realized, flexible capacity expansion can be realized only by adjusting the nodes stored in each group after the capacity expansion, and the migration quantity of data is reduced while the capacity expansion is carried out.
And S103, when the capacity expansion of the newly added node is determined, acquiring expanded first index information, wherein the first index information comprises the node serial numbers of the nodes after the capacity expansion and the grouping serial numbers corresponding to the node serial numbers, and the nodes corresponding to the grouping serial numbers are used for storing policy data corresponding to the grouping serial numbers.
In the capacity expansion process, unlike the existing node fission capacity expansion mode, the number of newly added nodes in the embodiment can be single or multiple. The first index information constructed in this embodiment indicates a correspondence between the node number of each node after capacity expansion and the packet number corresponding to each node number. The first index information may specifically be a configuration table as shown in fig. 3, or may be in the form of a data queue, an array, or the like. The present invention is not limited to any specific alternative form of the first index information, the second index information, the initial index information, and the like in the following embodiments.
In some embodiments, the capacity expansion may be first capacity expansion based on the number of the initial nodes, or may be second capacity expansion, third capacity expansion, or multiple capacity expansion, and step S103 (when determining that the new node expands, acquiring the first index information after capacity expansion) may specifically be adding the node number of the new node to the second index information before capacity expansion when determining that the new node expands, acquiring the first index information after capacity expansion, where the second index information is the initial index information, or the second index information is the first index information determined when the previous capacity expansion.
In some embodiments where the first expansion is performed based on the number of initial nodes, the existing node architecture lacks index information, and the embodiment may further include a process of first establishing the initial index information before acquiring the expanded first index information. And when the capacity is not expanded for the first time, flexible adjustment is performed depending on the index information. Specifically, for example, if the number of nodes before capacity expansion is determined to be consistent with the preset number of initial nodes, initial index information is determined according to the group serial numbers corresponding to the policy data and the number of nodes before capacity expansion, wherein the initial index information indicates the group serial numbers corresponding to the node serial numbers when the policy data is stored in the number of initial nodes before capacity expansion for the first time. As an optional implementation manner, the process of determining the initial index information according to the packet sequence number corresponding to each policy data and the number of nodes before capacity expansion may specifically be that the packet sequence number corresponding to each policy data is modulo the number of the initial nodes to obtain the node sequence number corresponding to each packet sequence number before capacity expansion; and then determining the node serial numbers of the nodes before capacity expansion and the group serial numbers corresponding to the node serial numbers as initial index information.
And S104, responding to the adjustment of the serial numbers of all nodes in the first index information to the serial numbers of the corresponding groups, and performing migration storage on the policy data in all nodes according to the adjusted first index information.
Before adjustment, the newly added node in the first index information has no corresponding packet sequence number. The adjustment rule may be to receive an externally input adjustment instruction to adjust the packet sequence number, or may automatically adjust according to a preset adjustment rule. In this embodiment, after the first index information is constructed, the packets stored in each node are adjusted correspondingly to the first index information. Referring to fig. 4, an example of packet sequence number adjustment is provided in an embodiment of the present invention. As shown in fig. 4, the packet numbers 0, 4, 8 are originally stored in the node 0, and after the adjustment, the packet numbers 0, 4, 8 are adjusted to correspond to the newly added node 4, and the policy data corresponding to the packet numbers 0, 4, 8 are also migrated to the node 4 for storage.
In some embodiments, a variety of conditions may be preset that trigger the expansion step. For example, the capacity expansion determination process is also performed before the following step S103. The capacity expansion judgment process first acquires, for example, stock information and/or read-write processing load information of each node. And determining that at least one node (such as node 0 shown in fig. 4) is in an overrun state according to the stock information and/or the read-write processing load information. The stock information is, for example, the total size of policy data stored in the node. Judging whether the data stored by each node reach the respective stock threshold, if the data exceeds the stock threshold, indicating that the data of the policy stored by the node is overlarge, and increasing the nodes to share the storage task is needed. The read-write processing load information is, for example, the read-write condition of policy data stored in the node. For example, short-term policy data is mainly stored in some nodes, and against the short-term policy data, there is a larger read-write requirement in the validity period of the short-term policy data, and the read-write operation may approach or exceed the processing capacity of the nodes, so that the node needs to be increased to transfer a part of policy data to share the read-write load. And sending out capacity expansion request information when at least one node is determined to be in an overrun state. The capacity expansion request information is, for example, a prompt message, or may be information triggering to automatically increase access of the node, which is not limited herein.
In the above embodiment of determining that at least one node is in an overrun state, step S104 may be, for example, adjusting one or more packet sequence numbers corresponding to the at least one node in the first index information to correspond to the newly added node sequence number; and according to the adjusted first index information, migrating and storing the policy data corresponding to the one or more packet sequence numbers into the newly added node. It can be understood that the packet sequence numbers stored in the nodes in the overrun state are migrated to the newly added nodes, so that storage balance is realized, the storage and/or read-write burden of the nodes in the overrun state is reduced, and the reliability of the whole storage system is improved.
According to the policy data migration storage method, the policy identification value corresponding to the policy data stored by each node before capacity expansion and the preset node number threshold are obtained, wherein the node serial number of the node stored by each policy data before primary capacity expansion is obtained by modulo the policy identification value of the policy data; respectively modulo the number threshold of the nodes by the policy identification value to obtain the grouping sequence numbers corresponding to the policy data, wherein the number threshold of the nodes is 2 of the number of the nodes before capacity expansion N Multiple, N is an integer greater than 0; when the capacity expansion of a newly added node is determined, acquiring expanded first index information, wherein the first index information comprises the node serial numbers of the nodes after the capacity expansion and the group serial numbers corresponding to the node serial numbers, and the nodes corresponding to the group serial numbers are used for storing policy data corresponding to the group serial numbers; in response to the adjustment of the corresponding grouping sequence numbers of the node sequence numbers in the first index information, migration storage is performed on the policy data in each node according to the adjusted first index information, so that the threshold value of the node quantity is modulo according to the policy identification value, the grouping sequence numbers which can be traced back to the node positions stored before the initial node is not expanded are determined for each policy data, thus the initial index information is established, the index information is adjusted, data migration storage is performed according to the adjusted index information, and expansion can be performed on the basis of the existing distributed storage systemAnd the number of newly added nodes and the data migration quantity required by each expansion are reduced, the dependence on direct hash modulo storage of the policy identification value in the existing expansion scheme is relieved, and the flexibility and the reliability of the policy data migration storage are improved.
On the basis of the above embodiment, after step S103 (when determining that the newly added node expands, acquiring the expanded first index information), the searching process of the policy data may further include: when a query request containing a to-be-checked policy identification value is obtained, the to-be-checked policy identification value is subjected to modulo operation on the node quantity threshold value to obtain a to-be-checked packet sequence number corresponding to the to-be-checked policy identification value; and searching the policy data according to the identification value of the policy to be checked in the node corresponding to the serial number of the packet to be checked.
On the basis of the above embodiment, after step S103 (when determining that the newly added node expands, the expanded first index information is acquired), the storage process of the new policy data may further include: when new policy data to be stored is acquired, acquiring a new policy identification value of the new policy data; the new policy identification value is subjected to module taking on the node quantity threshold value to obtain a grouping sequence number corresponding to the new policy data; acquiring a target node corresponding to the packet sequence number of the new policy data in the first index information; and storing the new policy data into the target node.
Referring to fig. 5, a block diagram of a policy data migration storage system provided by the present invention is shown. The system shown in fig. 5 may be, for example, a distributed data management system that performs the above-described method embodiment, or a background management server for controlling data reading and writing of each node, or the like. The policy data migration storage system shown in fig. 5 includes:
the obtaining module 51 is configured to obtain a policy identifier value corresponding to policy data stored in each node before capacity expansion, and a preset node number threshold, where a node serial number of a node stored in each policy data before primary capacity expansion is a value obtained by modulo the policy identifier value of the policy data with respect to the number of nodes before capacity expansion.
Grouping module 52 for respectively matching the policy identification values to the nodesObtaining the grouping sequence numbers corresponding to the policy data by modular number threshold, wherein the node number threshold is 2 of the node number before capacity expansion N Multiple, N is an integer greater than 0.
The capacity expansion module 53 is configured to obtain expanded first index information when it is determined that the newly added node expands capacity, where the first index information includes a node number of each node after expansion and a packet number corresponding to each node number, and a node corresponding to the packet number is configured to store policy data corresponding to the packet number.
And the processing module 54 is configured to respond to adjustment of the corresponding packet sequence number of each node sequence number in the first index information, and perform migration storage on the policy data in each node according to the adjusted first index information.
The policy data migration storage system of the embodiment shown in fig. 5 may be correspondingly used to perform the steps in the method embodiment shown in fig. 2, and the implementation principle and technical effects are similar, and are not repeated herein.
In some embodiments, the capacity expansion module 53 is configured to add, when determining that the newly added node expands, the node number of the newly added node to the second index information before expansion, and obtain the first index information after expansion, where the second index information is initial index information or the second index information is the first index information determined during the previous expansion.
In some embodiments, before adding the node number of the new node to the second index information before expansion, and before obtaining the first index information after expansion, the expansion module 53 is further configured to determine, if it is determined that the number of nodes before expansion is consistent with the number of preset initial nodes, the initial index information according to the group number corresponding to each policy data and the number of nodes before expansion, where the initial index information indicates the group number corresponding to each node number when each policy data is stored in the number of initial nodes before initial expansion.
In some embodiments, the capacity expansion module 53 is configured to modulo the number of the initial nodes by the packet sequence number corresponding to each policy data, to obtain a node sequence number corresponding to each packet sequence number before capacity expansion; and determining the node serial numbers of the nodes before capacity expansion and the group serial numbers corresponding to the node serial numbers as initial index information.
In some embodiments, the capacity expansion module 53 is further configured to obtain stock information and/or read-write processing load information of each node before obtaining the expanded first index information when determining that the newly added node expands capacity; determining that at least one node is in an overrun state according to the stock information and/or the read-write processing load information; and sending out capacity expansion request information.
Correspondingly, the processing module 54 is configured to adjust one or more packet sequence numbers corresponding to the at least one node in the first index information to correspond to the newly added node sequence number; and according to the adjusted first index information, migrating and storing the policy data corresponding to the one or more packet sequence numbers into the newly added node.
The processing module 54 is further configured to, when determining that the newly added node expands, obtain the expanded first index information, and then, when obtaining a query request including the identifier value of the policy to be checked, modulo the number threshold of the nodes by the identifier value of the policy to be checked, so as to obtain a serial number of the packet to be checked corresponding to the identifier value of the policy to be checked; and searching the policy data according to the identification value of the policy to be checked in the node corresponding to the serial number of the packet to be checked.
The processing module 54 is further configured to, when determining that the newly added node expands, obtain the expanded first index information, and then obtain a new policy identification value of the new policy data when obtaining the new policy data to be stored; the new policy identification value is subjected to module taking on the node quantity threshold value to obtain a grouping sequence number corresponding to the new policy data; acquiring a target node corresponding to the packet sequence number of the new policy data in the first index information; and storing the new policy data into the target node.
Referring to fig. 6, a schematic hardware structure of an electronic device according to an embodiment of the present invention is shown, where the electronic device 60 includes: a processor 61, a memory 62 and a computer program; wherein the method comprises the steps of
A memory 62 for storing the computer program, which memory may also be a flash memory (flash). Such as application programs, functional modules, etc. implementing the methods described above.
A processor 61 for executing the computer program stored in the memory to implement the steps in the policy data migration storage method. Reference may be made in particular to the description of the embodiments of the method described above.
Alternatively, the memory 62 may be separate or integrated with the processor 61.
When the memory 62 is a device separate from the processor 61, the electronic apparatus may further include:
a bus 63 for connecting the memory 62 and the processor 61.
The present invention also provides a computer readable storage medium having a computer program stored therein, which when executed by a processor is configured to implement the policy data migration storage method provided in the various embodiments described above.
The computer readable storage medium may be a computer storage medium or a communication medium. Communication media includes any medium that facilitates transfer of a computer program from one place to another. Computer storage media can be any available media that can be accessed by a general purpose or special purpose computer. For example, a computer-readable storage medium is coupled to the processor such that the processor can read information from, and write information to, the computer-readable storage medium. In the alternative, the computer-readable storage medium may be integral to the processor. The processor and the computer readable storage medium may reside in an application specific integrated circuit (Application Specific Integrated Circuits, ASIC for short). In addition, the ASIC may reside in a user device. The processor and the computer-readable storage medium may also reside as discrete components in a communication device. The computer readable storage medium may be Read Only Memory (ROM), random Access Memory (RAM), CD-ROM, magnetic tape, floppy disk, optical data storage device, etc.
The present invention also provides a program product comprising execution instructions stored in a computer-readable storage medium. The at least one processor of the device may read the execution instructions from the computer-readable storage medium, and execution of the execution instructions by the at least one processor causes the device to implement the policy data migration storage method provided by the various embodiments described above.
In the above embodiment of the electronic device, it should be understood that the processor may be a central processing unit (english: central Processing Unit, abbreviated as CPU), or may be other general purpose processors, digital signal processors (english: digital Signal Processor, abbreviated as DSP), application specific integrated circuits (english: application Specific Integrated Circuit, abbreviated as ASIC), or the like. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The steps of a method disclosed in connection with the present invention may be embodied directly in a hardware processor for execution, or in a combination of hardware and software modules in a processor for execution.
Finally, it should be noted that: the above embodiments are only for illustrating the technical solution of the present invention, and not for limiting the same; although the invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some or all of the technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit of the invention.
Claims (7)
1. A policy data migration storage method, comprising:
acquiring a policy identification value corresponding to policy data stored by each node before capacity expansion and a preset node quantity threshold, wherein the node serial number of the node stored by each policy data before primary capacity expansion is a value obtained by modulo the policy identification value of the policy data on the number of the nodes before capacity expansion;
respectively modulo the node quantity threshold value by the policy identification value to obtain eachThe number threshold of the nodes is 2 of the number of the nodes before capacity expansion N Multiple, N is an integer greater than 0;
if the number of the nodes before capacity expansion is determined to be consistent with the number of the preset initial nodes, initial index information is determined according to the group serial numbers corresponding to the policy data and the number of the nodes before capacity expansion, wherein the initial index information indicates the group serial numbers corresponding to the node serial numbers when the policy data is stored in the number of the initial nodes before the primary capacity expansion;
the determining initial index information according to the packet sequence number corresponding to each policy data and the number of nodes before capacity expansion includes:
the number of the initial nodes is modulo by the group serial number corresponding to each policy data, and the node serial number corresponding to each group serial number before capacity expansion is obtained;
determining the node serial numbers of the nodes before capacity expansion and the group serial numbers corresponding to the node serial numbers as initial index information;
when the capacity expansion of a newly added node is determined, acquiring expanded first index information, wherein the first index information comprises the node serial numbers of the nodes after the capacity expansion and the group serial numbers corresponding to the node serial numbers, and the nodes corresponding to the group serial numbers are used for storing policy data corresponding to the group serial numbers;
responding to the adjustment of the corresponding grouping sequence numbers of the sequence numbers of all nodes in the first index information, and performing migration storage on the policy data in all nodes according to the adjusted first index information;
before the capacity expansion of the newly added node is determined and the expanded first index information is obtained, the method further comprises the steps of:
acquiring stock information and/or read-write processing load information of each node;
determining that at least one node is in an overrun state according to the stock information and/or the read-write processing load information;
sending out capacity expansion request information;
the response to the adjustment of the serial numbers of the nodes in the first index information corresponding to the grouping serial numbers, the migration storage of the policy data in the nodes according to the adjusted first index information comprises the following steps:
one or more group serial numbers corresponding to the at least one node in the first index information are adjusted to correspond to the newly added node serial numbers;
and according to the adjusted first index information, migrating and storing the policy data corresponding to the one or more packet sequence numbers into the newly added node.
2. The method of claim 1, wherein the obtaining the expanded first index information when the newly added node is determined to be expanded comprises:
and when the capacity expansion of the newly added node is determined, adding the node serial number of the newly added node into second index information before capacity expansion, and acquiring first index information after capacity expansion, wherein the second index information is initial index information or the second index information is first index information determined during the previous capacity expansion.
3. The method according to any one of claims 1 to 2, further comprising, after obtaining the expanded first index information when determining that the newly added node expands,:
when a query request containing a to-be-checked policy identification value is obtained, the to-be-checked policy identification value is subjected to modulo operation on the node quantity threshold value to obtain a to-be-checked packet sequence number corresponding to the to-be-checked policy identification value;
and searching the policy data according to the identification value of the policy to be checked in the node corresponding to the serial number of the packet to be checked.
4. The method according to any one of claims 1 to 2, further comprising, after obtaining the expanded first index information when determining that the newly added node expands,:
when new policy data to be stored is acquired, acquiring a new policy identification value of the new policy data;
the new policy identification value is subjected to module taking on the node quantity threshold value to obtain a grouping sequence number corresponding to the new policy data;
acquiring a target node corresponding to the packet sequence number of the new policy data in the first index information;
and storing the new policy data into the target node.
5. A policy data migration storage system, comprising:
the system comprises an acquisition module, a storage module and a storage module, wherein the acquisition module is used for acquiring a policy identification value corresponding to policy data stored by each node before capacity expansion and a preset node quantity threshold value, wherein the node serial number of the node stored by each policy data before primary capacity expansion is a value obtained by modulo the policy identification value of the policy data on the node quantity before capacity expansion;
the grouping module is used for respectively modulo the number threshold of the nodes by the policy identification value to obtain the grouping sequence number corresponding to each policy data, wherein the number threshold of the nodes is 2 of the number of the nodes before capacity expansion N Multiple, N is an integer greater than 0;
the capacity expansion module is used for acquiring expanded first index information when the capacity expansion of the newly added node is determined, wherein the first index information comprises the node serial numbers of the nodes after the capacity expansion and the group serial numbers corresponding to the node serial numbers, and the nodes corresponding to the group serial numbers are used for storing policy data corresponding to the group serial numbers;
the processing module is used for responding to the adjustment of the corresponding grouping sequence numbers of the sequence numbers of all nodes in the first index information, and performing migration storage on the policy data in all nodes according to the adjusted first index information;
the capacity expansion module is further configured to determine initial index information according to the packet sequence numbers corresponding to the policy data and the number of nodes before expansion if the number of nodes before expansion is determined to be consistent with the number of preset initial nodes before adding the node sequence numbers of the newly added nodes to the second index information before expansion, and before acquiring the first index information after expansion, where the initial index information indicates the packet sequence numbers corresponding to the node sequence numbers when the policy data is stored in the number of initial nodes before initial expansion;
the capacity expansion module is specifically configured to modulo the number of the initial nodes by using a packet sequence number corresponding to each policy data, so as to obtain a node sequence number corresponding to each packet sequence number before capacity expansion; determining the node serial numbers of the nodes before capacity expansion and the group serial numbers corresponding to the node serial numbers as initial index information;
the capacity expansion module is used for acquiring stock information and/or read-write processing load information of each node before acquiring the expanded first index information when determining the capacity expansion of the newly added node; determining that at least one node is in an overrun state according to the stock information and/or the read-write processing load information; sending out capacity expansion request information;
the processing module is further used for adjusting one or more packet serial numbers corresponding to the at least one node in the first index information to correspond to the newly added node serial number; and according to the adjusted first index information, migrating and storing the policy data corresponding to the one or more packet sequence numbers into the newly added node.
6. An electronic device, comprising: a memory, a processor and a computer program stored in the memory, the processor running the computer program to perform the policy data migration storage method of any one of claims 1 to 4.
7. A computer readable storage medium, wherein a computer program is stored in the computer readable storage medium, which when executed by a processor is adapted to implement the policy data migration storage method of any one of claims 1 to 4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010108664.5A CN111274228B (en) | 2020-02-21 | 2020-02-21 | Policy data migration storage method, system, equipment and readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010108664.5A CN111274228B (en) | 2020-02-21 | 2020-02-21 | Policy data migration storage method, system, equipment and readable storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111274228A CN111274228A (en) | 2020-06-12 |
| CN111274228B true CN111274228B (en) | 2023-09-05 |
Family
ID=70997190
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010108664.5A Active CN111274228B (en) | 2020-02-21 | 2020-02-21 | Policy data migration storage method, system, equipment and readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111274228B (en) |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6880017B1 (en) * | 2000-03-20 | 2005-04-12 | International Business Machines Corporation | System and method for providing an adaptive streaming flow control mechanism between the TCP and IP layers of the TCP/IP suite of protocols |
| CN104378447A (en) * | 2014-12-03 | 2015-02-25 | 深圳市鼎元科技开发有限公司 | Non-migration distributed storage method and non-migration distributed storage system on basis of Hash ring |
| CN105653630A (en) * | 2015-12-25 | 2016-06-08 | 北京奇虎科技有限公司 | Data migration method and apparatus for distributed database |
| CN107562531A (en) * | 2016-06-30 | 2018-01-09 | 华为技术有限公司 | A kind of data balancing method and device |
| CN108009883A (en) * | 2017-11-30 | 2018-05-08 | 泰康保险集团股份有限公司 | Method and device for order processing |
| WO2018120939A1 (en) * | 2016-12-29 | 2018-07-05 | 华为技术有限公司 | Storage system expanded data migration method and storage system |
| CN109033278A (en) * | 2018-07-11 | 2018-12-18 | 江苏通付盾科技有限公司 | Data processing method, device, electronic equipment and computer storage medium |
| CN109213761A (en) * | 2018-08-06 | 2019-01-15 | 北京马上慧科技术有限公司 | Table method is divided in point library that a kind of spread exempts from migration |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9772775B2 (en) * | 2015-08-21 | 2017-09-26 | International Business Machines Corporation | Scalable and efficient access to and management of data and resources in a tiered data storage system |
-
2020
- 2020-02-21 CN CN202010108664.5A patent/CN111274228B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6880017B1 (en) * | 2000-03-20 | 2005-04-12 | International Business Machines Corporation | System and method for providing an adaptive streaming flow control mechanism between the TCP and IP layers of the TCP/IP suite of protocols |
| CN104378447A (en) * | 2014-12-03 | 2015-02-25 | 深圳市鼎元科技开发有限公司 | Non-migration distributed storage method and non-migration distributed storage system on basis of Hash ring |
| CN105653630A (en) * | 2015-12-25 | 2016-06-08 | 北京奇虎科技有限公司 | Data migration method and apparatus for distributed database |
| CN107562531A (en) * | 2016-06-30 | 2018-01-09 | 华为技术有限公司 | A kind of data balancing method and device |
| WO2018120939A1 (en) * | 2016-12-29 | 2018-07-05 | 华为技术有限公司 | Storage system expanded data migration method and storage system |
| CN108009883A (en) * | 2017-11-30 | 2018-05-08 | 泰康保险集团股份有限公司 | Method and device for order processing |
| CN109033278A (en) * | 2018-07-11 | 2018-12-18 | 江苏通付盾科技有限公司 | Data processing method, device, electronic equipment and computer storage medium |
| CN109213761A (en) * | 2018-08-06 | 2019-01-15 | 北京马上慧科技术有限公司 | Table method is divided in point library that a kind of spread exempts from migration |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111274228A (en) | 2020-06-12 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11586673B2 (en) | Data writing and reading method and apparatus, and cloud storage system | |
| CN107622091B (en) | Database query method and device | |
| CN109408590B (en) | Method, device and equipment for expanding distributed database and storage medium | |
| CN110427386B (en) | Data processing method, device and computer storage medium | |
| EP3640813B1 (en) | Cluster-based random walk method and apparatus | |
| CN111597040A (en) | Resource allocation method, device, storage medium and electronic equipment | |
| CN108399175B (en) | Data storage and query method and device | |
| CN111159130A (en) | Small file merging method and electronic equipment | |
| CN112269661A (en) | Partition migration method and device based on Kafka cluster | |
| CN105630419A (en) | Partition view sending method of resource pools and management node | |
| CN114721594A (en) | Distributed storage method, device, equipment and machine readable storage medium | |
| CN112860412B (en) | Service data processing method and device, electronic equipment and storage medium | |
| WO2021227789A1 (en) | Storage space allocation method and device, terminal, and computer readable storage medium | |
| CN111274228B (en) | Policy data migration storage method, system, equipment and readable storage medium | |
| CN113568877A (en) | File merging method and device, electronic equipment and storage medium | |
| CN111046004A (en) | Data file storage method, device, equipment and storage medium | |
| CN108846009B (en) | Copy data storage method and device in ceph | |
| CN115756955A (en) | Data backup and data recovery method and device and computer equipment | |
| CN112764897B (en) | Task request processing method, device and system and computer readable storage medium | |
| CN111464312B (en) | Method and device for processing account addresses in blockchain and electronic equipment | |
| CN110825521B (en) | Memory use management method and device and storage medium | |
| CN113419792A (en) | Event processing method and device, terminal equipment and storage medium | |
| CN113392131A (en) | Data processing method and device and computer equipment | |
| CN113761400A (en) | Access request forwarding method, device and equipment | |
| CN111817895B (en) | Master control node switching method, device, equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |