CN117076470A - Capacity expansion method, device, equipment, medium and program product for financial database - Google Patents
Capacity expansion method, device, equipment, medium and program product for financial database Download PDFInfo
- Publication number
- CN117076470A CN117076470A CN202311057220.3A CN202311057220A CN117076470A CN 117076470 A CN117076470 A CN 117076470A CN 202311057220 A CN202311057220 A CN 202311057220A CN 117076470 A CN117076470 A CN 117076470A
- Authority
- CN
- China
- Prior art keywords
- storage
- node
- capacity
- target
- service
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 50
- 239000012634 fragment Substances 0.000 claims abstract description 88
- 239000003795 chemical substances by application Substances 0.000 claims abstract description 72
- 230000008859 change Effects 0.000 claims abstract description 43
- 238000004590 computer program Methods 0.000 claims abstract description 26
- 238000011084 recovery Methods 0.000 claims abstract description 17
- 238000005192 partition Methods 0.000 claims description 52
- 238000005516 engineering process Methods 0.000 abstract description 9
- 238000010586 diagram Methods 0.000 description 6
- 230000005012 migration Effects 0.000 description 6
- 238000013508 migration Methods 0.000 description 6
- 238000012545 processing Methods 0.000 description 6
- 230000000903 blocking effect Effects 0.000 description 5
- 238000007726 management method Methods 0.000 description 4
- 238000004891 communication Methods 0.000 description 3
- 230000004044 response Effects 0.000 description 3
- 238000013500 data storage Methods 0.000 description 2
- 230000008569 process Effects 0.000 description 2
- 230000003068 static effect Effects 0.000 description 2
- OKTJSMMVPCPJKN-UHFFFAOYSA-N Carbon Chemical compound [C] OKTJSMMVPCPJKN-UHFFFAOYSA-N 0.000 description 1
- 230000009471 action Effects 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 229910021389 graphene Inorganic materials 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/211—Schema design and management
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application relates to a financial database expansion method, a financial database expansion device, a financial database expansion computer device, a financial database expansion storage medium and a financial database expansion computer program product, and relates to the fields of computer technology and financial science and technology. The application can realize the elastic expansion of the storage layer of the financial database and improve the stability and service performance of the whole cluster. The method comprises the following steps: according to the current configuration information sent by the front-end console, newly adding at least one capacity-expansion storage node and determining capacity-expansion storage fragments matched with the capacity-expansion storage node; updating local route information according to the route change information; under the condition that the local routing information updating is completed, sending a routing information updating instruction to each proxy node so as to enable each proxy node to suspend original writing service and update the routing information; and sending a service recovery instruction to the target agent node which has completed updating in each agent node, so that the target agent node executes corresponding read service according to the updated routing information and executes corresponding write service according to the updated routing information.
Description
Technical Field
The present application relates to the field of computer technology and financial science and technology, and in particular, to a method, an apparatus, a computer device, a storage medium and a computer program product for expanding a financial database.
Background
With the continuous development of computer and big data technologies, banking institutions generate hundreds of millions of financial data each day, and the generated financial data are typically stored in a financial database for subsequent viewing or use.
In the conventional technology, when a financial database is expanded, data migration is required for data stored therein. However, the external service logic and the data migration logic of the financial database are both the same main thread and are responsible for execution, so that the problems of resource preemption, service blocking and the like can occur in the data migration process, and the overall stability and service performance of the cluster are reduced.
Disclosure of Invention
In view of the foregoing, it is desirable to provide a method, apparatus, computer device, computer-readable storage medium, and computer program product for expanding a financial database.
In a first aspect, the present application provides a method for expanding a financial database. The method comprises the following steps:
Responding to a capacity expansion instruction of a financial database, adding at least one capacity expansion storage node according to current configuration information sent by a front-end control console, and determining capacity expansion storage fragments matched with the capacity expansion storage nodes to obtain corresponding route change information; the capacity expansion storage partition comprises an original storage node and at least one capacity expansion storage node;
updating the local routing information according to the routing change information;
under the condition that the local routing information updating is completed, sending a routing information updating instruction to each proxy node so as to enable each proxy node to suspend original writing service and update routing information;
and sending a service recovery instruction to the target agent node which has completed updating in the agent nodes, so that the target agent node executes corresponding read service according to the updated routing information and executes corresponding write service according to the updated routing information.
In one embodiment, the adding at least one expansion storage node according to the current configuration information sent by the front-end console and determining the expansion storage fragments matched with the expansion storage node includes:
determining the newly increased number of the capacity expansion storage nodes according to the current configuration information; and matching the capacity-expansion storage nodes with corresponding storage fragments according to a preset rule according to the newly-increased number of the capacity-expansion storage nodes and the arrangement sequence of the storage fragments to obtain the capacity-expansion storage fragments.
In one embodiment, the target proxy node executes a corresponding read service according to the updated routing information, including:
when the read request data is character string type data, the target agent node determines a target storage fragment according to the read request; forwarding the read request to all storage nodes in the capacity-expansion storage partition under the condition that the target storage partition is the capacity-expansion storage partition; and sending the target value to the metadata server under the condition that the target value exists in the returned result of any storage node.
In one embodiment, the target proxy node executes a corresponding read service according to the updated routing information, and further includes:
when the read request data is set type data, the target proxy node determines a target storage fragment according to the read request; when the target storage fragment is the capacity-expansion storage fragment, forwarding the read request to all storage nodes in the capacity-expansion storage fragment simultaneously; and receiving and combining the returned results of the storage nodes.
In one embodiment, the executing the corresponding write service according to the updated routing information includes:
When the writing request data is character string type data, the target agent node determines a target storage fragment according to the writing request; and executing the writing service corresponding to the parameter information according to the parameter information in the writing request under the condition that the target storage fragment is the capacity expansion storage fragment.
In one embodiment, the executing the corresponding write service according to the updated routing information further includes:
when the write request data is set type data, the target proxy node determines a target storage fragment according to the write request; and under the condition that the target storage fragment is the capacity-expansion storage fragment, inquiring whether a storage node which stores the target value in the writing request exists in the capacity-expansion storage fragment, and executing the writing service corresponding to the target set type according to the inquiring result.
In one embodiment, the respective proxy nodes suspend the original write service and update the routing information, including:
each proxy node responds to the routing information updating instruction and pauses the original writing service; acquiring the route change information from a metadata server, and updating the route information of the proxy node according to the route change information; sending updated feedback information to the metadata server under the condition that the routing information of the proxy node is updated;
Before sending the service restoration instruction to the target agent node which has completed updating in the agent nodes, the method further comprises:
receiving and counting the updated feedback information sent by the proxy node; and identifying the agent node which completes updating within a preset time window as the target agent node according to the statistical result.
In a second aspect, the application further provides a capacity expansion device of the financial database. The device comprises:
the node newly-added module is used for responding to the capacity-expanding instruction of the financial database, newly adding at least one capacity-expanding storage node according to the current configuration information sent by the front-end control console, and determining the capacity-expanding storage fragments matched with the capacity-expanding storage node to obtain corresponding route change information; the capacity expansion storage partition comprises an original storage node and at least one capacity expansion storage node;
the route updating module is used for updating the local route information according to the route change information;
the instruction sending module is used for sending routing information updating instructions to each proxy node under the condition that the local routing information updating is completed, so that each proxy node pauses the original writing service and updates the routing information;
And the service recovery module is used for sending a service recovery instruction to the target agent node which has completed updating in the agent nodes so that the target agent node executes corresponding read service according to the updated route information and executes corresponding write service according to the updated route information.
In a third aspect, the present application also provides a computer device. The computer device comprises a memory storing a computer program and a processor which when executing the computer program performs the steps of:
responding to a capacity expansion instruction of a financial database, adding at least one capacity expansion storage node according to current configuration information sent by a front-end control console, and determining capacity expansion storage fragments matched with the capacity expansion storage nodes to obtain corresponding route change information; the capacity expansion storage partition comprises an original storage node and at least one capacity expansion storage node; updating the local routing information according to the routing change information; under the condition that the local routing information updating is completed, sending a routing information updating instruction to each proxy node so as to enable each proxy node to suspend original writing service and update routing information; and sending a service recovery instruction to the target agent node which has completed updating in the agent nodes, so that the target agent node executes corresponding read service according to the updated routing information and executes corresponding write service according to the updated routing information.
In a fourth aspect, the present application also provides a computer-readable storage medium. The computer readable storage medium having stored thereon a computer program which when executed by a processor performs the steps of:
responding to a capacity expansion instruction of a financial database, adding at least one capacity expansion storage node according to current configuration information sent by a front-end control console, and determining capacity expansion storage fragments matched with the capacity expansion storage nodes to obtain corresponding route change information; the capacity expansion storage partition comprises an original storage node and at least one capacity expansion storage node; updating the local routing information according to the routing change information; under the condition that the local routing information updating is completed, sending a routing information updating instruction to each proxy node so as to enable each proxy node to suspend original writing service and update routing information; and sending a service recovery instruction to the target agent node which has completed updating in the agent nodes, so that the target agent node executes corresponding read service according to the updated routing information and executes corresponding write service according to the updated routing information.
In a fifth aspect, the present application also provides a computer program product. The computer program product comprises a computer program which, when executed by a processor, implements the steps of:
Responding to a capacity expansion instruction of a financial database, adding at least one capacity expansion storage node according to current configuration information sent by a front-end control console, and determining capacity expansion storage fragments matched with the capacity expansion storage nodes to obtain corresponding route change information; the capacity expansion storage partition comprises an original storage node and at least one capacity expansion storage node; updating the local routing information according to the routing change information; under the condition that the local routing information updating is completed, sending a routing information updating instruction to each proxy node so as to enable each proxy node to suspend original writing service and update routing information; and sending a service recovery instruction to the target agent node which has completed updating in the agent nodes, so that the target agent node executes corresponding read service according to the updated routing information and executes corresponding write service according to the updated routing information.
According to the capacity expansion method, the capacity expansion device, the computer equipment, the storage medium and the computer program product of the financial database, corresponding route change information is obtained by responding to the capacity expansion instruction of the financial database, adding at least one capacity expansion storage node according to the current configuration information sent by the front-end console and determining the capacity expansion storage fragments matched with the capacity expansion storage node; the capacity expansion storage partition comprises an original storage node and at least one capacity expansion storage node; updating the local routing information according to the routing change information; under the condition that the local routing information updating is completed, sending a routing information updating instruction to each proxy node so as to enable each proxy node to suspend original writing service and update the routing information; and sending a service recovery instruction to the target agent node which has completed updating in each agent node, so that the target agent node executes corresponding read service according to the updated routing information and executes corresponding write service according to the updated routing information. In this way, by utilizing the technical means of cluster centralized management and proxy layer self-adaptive route forwarding request, the elastic capacity expansion of the storage layer of the financial database is realized, so that the data migration of the data stored in the storage layer is avoided, the problems of resource preemption, service blocking and the like in the traditional technology are solved, and the overall stability and service performance of the cluster are effectively improved.
Drawings
FIG. 1 is a diagram of an application environment of a method for expanding a financial database in one embodiment;
FIG. 2 is a flow chart of a method for expanding a financial database according to one embodiment;
FIG. 3 is a schematic diagram of a storage layer corresponding to a newly added capacity expansion storage node in an embodiment;
FIG. 4 is a flow chart of a string type read request step in one embodiment;
FIG. 5 is a flow chart of a method for expanding a financial database according to an embodiment;
FIG. 6 is a flowchart of another embodiment of a method for expanding a financial database;
FIG. 7 is a block diagram illustrating an apparatus for expanding a financial database according to an embodiment;
fig. 8 is an internal structural diagram of a computer device in one embodiment.
Detailed Description
The present application will be described in further detail with reference to the drawings and examples, in order to make the objects, technical solutions and advantages of the present application more apparent. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the application.
It should be noted that, the client information (including, but not limited to, client device information, client personal information, etc.) and the data (including, but not limited to, data for analysis, stored data, presented data, etc.) related to the present application are information and data authorized by the client or sufficiently authorized by each party, and the collection, use and processing of the related data need to comply with the relevant laws and regulations and standards of the relevant country and region.
The method for expanding the financial database provided by the embodiment of the application can be applied to an application environment shown in fig. 1. Wherein,
the terminal 102 communicates with the server 104 via a network. The data storage system may store data that the server 104 needs to process. The data storage system may be integrated on the server 104 or may be located on a cloud or other network server. The front-end console 101 communicates with the metadata server 102 through a network, the proxy node 1 and the proxy node 2 communicate with the metadata server 102 through a network, respectively, and the storage node 1, the storage node 2, and the storage node 3 communicate with the proxy node 1 and the proxy node 2 through a network, respectively.
Specifically, the metadata server 102 responds to the capacity expansion instruction of the financial database, and according to the current configuration information sent by the front-end console 101, newly adds at least one capacity expansion storage node and determines capacity expansion storage fragments matched with the capacity expansion storage node, so as to obtain corresponding route change information; the capacity expansion storage partition comprises an original storage node and at least one capacity expansion storage node; the metadata server 102 updates the local routing information according to the routing change information; under the condition that the local routing information updating is completed, sending a routing information updating instruction to each proxy node so as to enable each proxy node to suspend original writing service and update the routing information; the metadata server 102 transmits a service restoration instruction to the target agent node that has completed updating among the agent nodes, so that the target agent node performs a corresponding read service according to the updated routing information and a corresponding write service according to the updated routing information.
In the application environment shown in fig. 1, the front-end console 101 may be a terminal or a server, and the metadata server 102 may be implemented as a stand-alone server or a server cluster composed of a plurality of servers.
In one embodiment, as shown in fig. 2, a method for expanding a financial database is provided, and the method is applied to the metadata server 102 in fig. 1 for illustration, and includes the following steps:
step S201, responding to the capacity expansion instruction of the financial database, newly adding at least one capacity expansion storage node according to the current configuration information sent by the front-end control console, and determining the capacity expansion storage fragments matched with the capacity expansion storage nodes to obtain the corresponding route change information; the capacity-expansion storage partition comprises an original storage node and at least one capacity-expansion storage node.
The financial database may be a Key-value database, a database storing data in Key-value pairs, such as a dis Cluster database.
The front-end control console is used for controlling and maintaining the clusters, providing a man-machine interface, controlling pages by the front-end, and the like.
Wherein the storage layer is divided into 1024 slots (0-1023), and when the cluster is created, the slots are evenly distributed to each storage partition in the system; in an initial state, each slot corresponds to 1 storage node, when the load is increased to trigger elastic capacity expansion, the original storage node and the capacity expansion storage node are simultaneously responsible for overload slots by newly increasing the capacity expansion storage nodes and adjusting route information, and newly-increased data in the slots are written into the capacity expansion storage nodes.
Specifically, the metadata server 102 responds to the capacity expansion instruction of the financial database, acquires current configuration information sent by the front-end console, adds at least one capacity expansion storage node according to the current configuration information, and matches the capacity expansion storage nodes with corresponding capacity expansion storage fragments according to preset rules to obtain corresponding route change information; the capacity-expansion storage partition comprises an original storage node and at least one capacity-expansion storage node.
Step S202, the local route information is updated according to the route change information.
The route change information may be corresponding route information after the capacity expansion of the financial database, and the local route information refers to route information of the metadata server.
Specifically, the metadata server 102 updates the local routing information according to the routing change information.
Step S203, in the case where the local routing information update is completed, a routing information update instruction is sent to each proxy node, so that each proxy node suspends the original write service and updates the routing information.
The agent node is positioned at the agent layer and used for acquiring the route information from the metadata server and storing the route information in the local area; when a client request is received, it is sent to the correct storage node according to the routing information.
Specifically, the metadata server 102 generates a routing information update instruction in the case where the local routing information update is completed, and transmits the routing information update instruction to each proxy node to cause each proxy node to suspend the original write service and update the routing information.
Step S204, sending a service recovery instruction to the target agent node which has completed updating in each agent node, so that the target agent node executes corresponding read service according to the updated route information and executes corresponding write service according to the updated route information.
Specifically, the metadata server 102 receives and counts updated feedback information sent by the proxy node, and identifies the proxy node that completes updating within a preset time window as a target proxy node according to the statistics result; and sending a service recovery instruction to a target proxy node in each proxy node so that the target proxy node executes corresponding read service according to the updated route information and executes corresponding write service according to the updated route information.
In the capacity expansion method of the financial database, corresponding route change information is obtained by responding to the capacity expansion instruction of the financial database, adding at least one capacity expansion storage node according to the current configuration information sent by the front-end console, and determining the capacity expansion storage fragments matched with the capacity expansion storage nodes; the capacity expansion storage partition comprises an original storage node and at least one capacity expansion storage node; updating the local routing information according to the routing change information; under the condition that the local routing information updating is completed, sending a routing information updating instruction to each proxy node so as to enable each proxy node to suspend original writing service and update the routing information; and sending a service recovery instruction to the target agent node which has completed updating in each agent node, so that the target agent node executes corresponding read service according to the updated routing information and executes corresponding write service according to the updated routing information. In this way, by utilizing the technical means of cluster centralized management and proxy layer self-adaptive route forwarding request, the elastic capacity expansion of the storage layer of the financial database is realized, so that the data migration of the data stored in the storage layer is avoided, the problems of resource preemption, service blocking and the like in the traditional technology are solved, and the overall stability and service performance of the cluster are effectively improved.
In one embodiment, according to current configuration information sent by a front-end console, at least one expansion storage node is newly added and expansion storage fragments matched with the expansion storage nodes are determined, and the method specifically comprises the following steps:
determining the newly increased number of the capacity expansion storage nodes according to the current configuration information; and matching the capacity-expansion storage nodes with the corresponding storage fragments according to a preset rule according to the newly-increased number of the capacity-expansion storage nodes and the arrangement sequence of the storage fragments to obtain the capacity-expansion storage fragments.
The capacity expansion storage partition refers to a storage partition comprising an original storage node and at least one capacity expansion storage node; as shown in fig. 3, a schematic diagram corresponding to a storage layer after a new capacity expansion storage node is provided in an embodiment.
Specifically, the metadata server 102 determines, according to the current configuration information, the newly increased number of capacity expansion storage nodes; and matching the capacity-expansion storage nodes with the corresponding storage fragments according to a preset rule according to the newly-increased number of the capacity-expansion storage nodes and the arrangement sequence of the storage fragments to obtain the capacity-expansion storage fragments.
In this embodiment, according to the newly increased number of the capacity expansion storage nodes and the arrangement sequence of each storage fragment, the capacity expansion storage nodes are matched with the corresponding storage fragments according to a preset rule; thereby accurately and orderly obtaining the capacity-expansion storage fragments and realizing the elastic capacity expansion of the storage layer of the financial database.
In one embodiment, the target proxy node executes a corresponding read service according to the updated routing information, and specifically includes the following steps:
when the read request data is character string type data, the target proxy node determines a target storage fragment according to the read request; forwarding a read request to all storage nodes in the capacity-expanding storage partition under the condition that the target storage partition is the capacity-expanding storage partition; and sending the target value to the metadata server when the target value exists in the returned result of any storage node.
The character string is a common data type, is widely applied in computer programming, is a sequence of characters, and can contain letters, numbers, symbols and the like.
In which, as shown in fig. 4, a flow chart of a string type read request step in one embodiment is provided.
Specifically, under the condition that the target agent node recognizes that the read request data is character string type data, determining a target storage fragment according to the read request; forwarding a read request to all storage nodes in the capacity-expanding storage partition under the condition that the target storage partition is the capacity-expanding storage partition; and sending the target value to the metadata server when the target value exists in the returned result of any storage node.
For example, the target agent node calculates a slot value according to the keyword in the request, finds all storage nodes corresponding to the slot according to the local routing information, and sends the request to all storage nodes; the target agent node receives the returned results of all the storage nodes, and returns the null to the client if all the returned results of all the storage nodes are null; and if the target value exists in the returned result of a certain storage node, returning the target value to the client as a result.
In this embodiment, the read request is forwarded to all storage nodes in the capacity-expanding storage partition under the condition that the target storage partition is the capacity-expanding storage partition; under the condition that a target value exists in a return result of any storage node, the target value is sent to a metadata server; therefore, corresponding read service is accurately executed according to the updated route information, and the stability and service performance of the whole cluster are improved.
In one embodiment, the target proxy node executes the corresponding read service according to the updated routing information, and further includes the following steps:
when the read request data is set type data, the target proxy node determines a target storage fragment according to the read request; when the target storage partition is a capacity-expansion storage partition, forwarding a read request to all storage nodes in the capacity-expansion storage partition simultaneously; and receiving and combining the returned results of the storage nodes.
Wherein the collection type data includes, but is not limited to set, zset, list, hash, each key comprising a plurality of elements. After the capacity expansion action, all storage nodes corresponding to the slots may contain elements of a certain keyword, so special processing is required.
Specifically, when the target agent node identifies that the read request data is set type data, determining a target storage fragment according to the read request; when the target storage partition is a capacity-expansion storage partition, forwarding a read request to all storage nodes in the capacity-expansion storage partition simultaneously; and receiving and combining the returned results of the storage nodes to obtain a final returned result.
For example, the target agent node calculates slots according to keys, simultaneously executes read requests (concurrency) on all storage nodes corresponding to the slots, and then merges the results returned by the storage nodes; wherein, set type: the Redis node returns the results (a, b, c), (e, f), and the combined result is (a, b, c, e, f); hash type: the Redis node returns the results (f 2: vs2, f3: vs 3), (f 1: vt1, f4: vt 4), and the combined result is (f 1: vt1, f4: vt4, f2: vs2, f3: vs 3); the merging mode of the Zset type is similar to set; the List type merge is similar to set.
In this embodiment, in the case that the target storage partition is a capacity-expansion storage partition, the read request is forwarded to all storage nodes in the capacity-expansion storage partition at the same time; receiving and combining the returned results of all the storage nodes; therefore, corresponding read service is executed pertinently according to the updated route information accurately, and the practicality, flexibility and service performance of the whole cluster are improved.
In one embodiment, the corresponding write service is executed according to the updated routing information, and specifically includes the following steps:
when the writing request data is character string type data, the target agent node determines a target storage fragment according to the writing request; and executing the writing service corresponding to the parameter information according to the parameter information in the writing request when the target storage fragment is the capacity expansion storage fragment.
Specifically, when the target agent node identifies that the write request data is character string type data, determining a target storage fragment according to the write request; and executing the writing service corresponding to the parameter information according to whether the writing request carries the related parameter information or not under the condition that the target storage fragment is the capacity expansion storage fragment.
For example, the target agent node calculates a slot value according to the key word, and determines all storage nodes according to the slot value:
(1) If there is only one storage node, the request is forwarded to the node, and finally the execution result of the storage node is returned to the client, and the processing is finished.
(2) Otherwise, executing corresponding logic according to the specific command:
(1) for Set commands that do not carry the NX and XX parameters:
1) In addition to the latest storage node, a Set command is sent to the other storage nodes, accompanied by an XX parameter (updates are written only if a key exists).
2) If a certain storage node returns success, the success is returned to the client, and the processing is ended.
3) And if the returned result contains success, returning success to the client.
4) If all the returned values of the operations are failed, the write operation is executed to the latest storage node, and the result is returned to the client.
(2) For a Set command carrying an NX parameter (added only if the key is not present):
1) And sending an exist command to all the nodes concurrently, and if the key words to be set exist in the nodes, returning failure to the client.
2) Otherwise, the Setnx command is sent to the latest node, and the execution result is returned to the client.
(3) For Set commands carrying XX parameters:
1) And sending a Set command to all storage nodes corresponding to the keywords, and attaching XX parameters.
2) If a certain storage node returns success, the success is returned to the client, otherwise, failure is returned to the client.
In this embodiment, in the case where the target storage partition is an expansion storage partition, according to the parameter information in the write request, the write service corresponding to the parameter information is executed pertinently; therefore, the practicality, flexibility and service performance of the whole cluster are improved.
In one embodiment, the corresponding write service is executed according to the updated routing information, and the method further includes the following steps:
when the write request data is set type data, the target proxy node determines a target storage fragment according to the write request; and under the condition that the target storage fragment is the capacity expansion storage fragment, inquiring whether a storage node for storing the target value in the writing request exists in the capacity expansion storage fragment, and executing the writing service corresponding to the target set type according to the inquiring result.
Specifically, when the target agent node identifies that the write request data is set type data, determining a target storage fragment according to the write request; under the condition that the target storage fragment is a capacity expansion storage fragment, inquiring whether a storage node for storing a target value in a writing request exists in the capacity expansion storage fragment or not, and obtaining a corresponding inquiring result; and executing the writing service corresponding to the target set type according to the query result.
For example, for all set-type write requests, a query request is first initiated to a storage node to which a slot corresponding to a key belongs, and whether a target value in the write request exists in the storage node is confirmed.
(1) For set type, hash type, zset type:
(1) if a certain storage node returns a presence result, forwarding the write request to the storage node;
(2) if no storage node returns the presence result, the write request is forwarded to the latest storage node.
(2) For List types, LPUSH, BLPOP are not supported, because the principle of the method is not to add data to the old node,
(1) firstly, inquiring the storage nodes with the key words in all the storage nodes in charge of the slot.
(2) For RPush commands: the write request is forwarded to the latest storage node.
(3) The LPop command forwards the request to the first storage node that owns the key, and returns a null if no storage node contains the key.
(4) RPop command: the request is forwarded to the last storage node that owns the key and if no storage node contains the key, a null is returned.
(5) BRPop command: the request is forwarded to the last storage node that owns the key, and if no storage node contains the key, the request is sent to the last node.
In this embodiment, by querying whether a storage node storing a target value in a write request exists in a capacity-expanding storage partition when the target storage partition is a capacity-expanding storage partition, according to a query result, a write service corresponding to a target set type is pertinently executed; therefore, the practicality, flexibility and service performance of the whole cluster are improved.
In one embodiment, each proxy node pauses the original write service and updates the routing information, specifically comprising the steps of:
each proxy node responds to the routing information updating instruction and pauses the original writing service; acquiring route change information from a metadata server, and updating the route information of the proxy node according to the route change information; under the condition that the update of the routing information of the proxy node is completed, the updated feedback information is sent to a metadata server;
before sending the service restoration instruction to the target agent node which has completed updating in each agent node, the method further comprises the following steps:
receiving and counting updated feedback information sent by the proxy node; and identifying the agent node which completes updating within a preset time window as a target agent node according to the statistical result.
The preset time window may be a time period set according to the actual application requirement, such as 3 seconds, 5 seconds, and the like.
Specifically, each proxy node pauses the original write service in response to the routing information update instruction; acquiring route change information from a metadata server, and updating the route information of the proxy node according to the route change information; sending updated feedback information to the metadata server under the condition that the update of the routing information of the proxy node is identified to be completed; the metadata server receives and counts updated feedback information sent by the proxy node to obtain a corresponding statistical result; and identifying the agent node which completes updating within a preset time window as a target agent node according to the statistical result.
In this embodiment, the updated feedback information sent by the proxy node is received and counted to obtain a corresponding statistical result, so that the target proxy node which completes updating within the preset time window is accurately identified according to the statistical result, and effective guarantee is provided for realizing elastic capacity expansion of the storage layer of the financial database.
In one embodiment, as shown in fig. 5, a method for expanding a financial database in an embodiment is provided, which specifically includes the following steps:
Step S501, in response to the capacity expansion instruction of the financial database, determines the new number of capacity expansion storage nodes according to the current configuration information sent by the front-end console.
Step S502, according to the newly increased number of the capacity expansion storage nodes and the arrangement sequence of the storage fragments, the capacity expansion storage nodes are matched with the corresponding storage fragments according to a preset rule, so that the capacity expansion storage fragments are obtained, and the corresponding route change information is obtained.
Step S503, updating the local routing information according to the routing change information.
Step S504, when the local route information updating is completed, route information updating instructions are sent to each proxy node so that each proxy node pauses the original writing service and updates the route information.
Step S505, receiving and counting updated feedback information sent by the proxy node; and identifying the agent node which completes updating within a preset time window as a target agent node according to the statistical result.
Step S506, sending a service recovery instruction to the target agent node which has completed updating in each agent node, so that the target agent node executes corresponding read service according to the updated route information and executes corresponding write service according to the updated route information.
In the capacity expansion method of the financial database, the elastic capacity expansion of the storage layer of the financial database is realized by utilizing the technical means of cluster centralized management and proxy layer self-adaptive route forwarding request, so that the data stored in the storage layer is prevented from being migrated, the problems of resource preemption, service blocking and the like in the traditional technology are solved, and the overall stability and service performance of the cluster are effectively improved.
In order to more clearly illustrate the capacity expansion method of the financial database provided by the embodiment of the application, a specific embodiment is used for specifically explaining the capacity expansion method of the financial database. In one embodiment, as shown in fig. 6, another method for expanding a financial database is provided, which specifically includes the following steps:
step one: the new current configuration information is updated to the metadata server via the front-end console.
Step two: the metadata server acquires the route change information and updates the local route information.
Step three: the metadata server informs all proxy nodes to pause writing service, and the reading service is normal.
Step four: when all proxy nodes reply that the metadata server has completed updating, notifying all proxy nodes to start writing service.
Step five: when part of proxy nodes do not reply to the metadata server within a preset time window, restarting the non-replied proxy nodes, and then notifying other proxy nodes to start writing service.
The beneficial effects brought by the embodiment are as follows:
the problems of resource preemption, service blocking and the like in the prior financial database during capacity expansion are solved, the elastic capacity expansion of the storage layer is realized through cluster centralized management and proxy layer self-adaptive route forwarding request, data migration is not needed, and the cluster service performance and stability are improved.
It should be understood that, although the steps in the flowcharts related to the embodiments described above are sequentially shown as indicated by arrows, these steps are not necessarily sequentially performed in the order indicated by the arrows. The steps are not strictly limited to the order of execution unless explicitly recited herein, and the steps may be executed in other orders. Moreover, at least some of the steps in the flowcharts described in the above embodiments may include a plurality of steps or a plurality of stages, which are not necessarily performed at the same time, but may be performed at different times, and the order of the steps or stages is not necessarily performed sequentially, but may be performed alternately or alternately with at least some of the other steps or stages.
Based on the same inventive concept, the embodiment of the application also provides a capacity expansion device of the financial database for realizing the capacity expansion method of the financial database. The implementation of the solution provided by the device is similar to the implementation described in the above method, so the specific limitation in the embodiments of the capacity expansion device for one or more financial databases provided below may refer to the limitation of the capacity expansion method for a financial database in the above description, and will not be repeated here.
In one embodiment, as shown in fig. 7, there is provided a capacity expansion device of a financial database, including:
the node adding module 701 is configured to respond to a capacity expansion instruction of the financial database, add at least one capacity expansion storage node according to current configuration information sent by the front-end console, and determine capacity expansion storage fragments matched with the capacity expansion storage node, so as to obtain corresponding route change information; the capacity expansion storage partition comprises an original storage node and at least one capacity expansion storage node;
a route update module 702, configured to update local route information according to route change information;
an instruction sending module 703, configured to send a routing information update instruction to each proxy node, in a case where the update of the local routing information is completed, so that each proxy node pauses the original write service and updates the routing information;
And the service recovery module 704 is configured to send a service recovery instruction to the target proxy node that has completed updating in each proxy node, so that the target proxy node executes a corresponding read service according to the updated routing information and executes a corresponding write service according to the updated routing information.
In one embodiment, the node adding module 701 is further configured to determine, according to the current configuration information, the number of added expansion storage nodes; and matching the capacity-expansion storage nodes with the corresponding storage fragments according to a preset rule according to the newly-increased number of the capacity-expansion storage nodes and the arrangement sequence of the storage fragments to obtain the capacity-expansion storage fragments.
In one embodiment, the capacity expansion device of the financial database further includes a first reading module, configured to determine, according to the read request, a target storage fragment when the read request data is character string type data; forwarding a read request to all storage nodes in the capacity-expanding storage partition under the condition that the target storage partition is the capacity-expanding storage partition; and sending the target value to the metadata server when the target value exists in the returned result of any storage node.
In one embodiment, the capacity expansion device of the financial database further comprises a second reading module, configured to determine, according to the read request, a target storage fragment when the read request data is set type data; when the target storage partition is a capacity-expansion storage partition, forwarding a read request to all storage nodes in the capacity-expansion storage partition simultaneously; and receiving and combining the returned results of the storage nodes.
In one embodiment, the capacity expansion device of the financial database further includes a first writing module, configured to determine, according to the writing request, a target storage partition when the writing request data is character string type data; and executing the writing service corresponding to the parameter information according to the parameter information in the writing request when the target storage fragment is the capacity expansion storage fragment.
In one embodiment, the capacity expansion device of the financial database further comprises a second writing module, configured to determine, according to the writing request, a target storage fragment when the writing request data is set type data; and under the condition that the target storage fragment is the capacity expansion storage fragment, inquiring whether a storage node for storing the target value in the writing request exists in the capacity expansion storage fragment, and executing the writing service corresponding to the target set type according to the inquiring result.
In one embodiment, the capacity expansion device of the financial database further comprises a route update module, configured to suspend the original writing service in response to the route information update instruction; acquiring route change information from a metadata server, and updating the route information of the proxy node according to the route change information; under the condition that the update of the routing information of the proxy node is completed, the updated feedback information is sent to a metadata server; the capacity expansion device of the financial database further comprises a feedback statistics module, which is used for receiving and counting updated feedback information sent by the proxy node; and identifying the agent node which completes updating within a preset time window as a target agent node according to the statistical result.
The above-mentioned various modules in the capacity expansion device of the financial database may be implemented in whole or in part by software, hardware, and a combination thereof. The above modules may be embedded in hardware or may be independent of a processor in the computer device, or may be stored in software in a memory in the computer device, so that the processor may call and execute operations corresponding to the above modules.
In one embodiment, a computer device is provided, which may be a server, and the internal structure of which may be as shown in fig. 8. The computer device includes a processor, a memory, an Input/Output interface (I/O) and a communication interface. The processor, the memory and the input/output interface are connected through a system bus, and the communication interface is connected to the system bus through the input/output interface. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The database of the computer device is used for storing data such as routing information. The input/output interface of the computer device is used to exchange information between the processor and the external device. The communication interface of the computer device is used for communicating with an external terminal through a network connection. The computer program, when executed by a processor, implements a method of expanding a financial database.
It will be appreciated by those skilled in the art that the structure shown in FIG. 8 is merely a block diagram of some of the structures associated with the present inventive arrangements and is not limiting of the computer device to which the present inventive arrangements may be applied, and that a particular computer device may include more or fewer components than shown, or may combine some of the components, or have a different arrangement of components.
In an embodiment, there is also provided a computer device comprising a memory and a processor, the memory having stored therein a computer program, the processor implementing the steps of the method embodiments described above when the computer program is executed.
In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored which, when executed by a processor, carries out the steps of the method embodiments described above.
In an embodiment, a computer program product is provided, comprising a computer program which, when executed by a processor, implements the steps of the method embodiments described above.
Those skilled in the art will appreciate that implementing all or part of the above described methods may be accomplished by way of a computer program stored on a non-transitory computer readable storage medium, which when executed, may comprise the steps of the embodiments of the methods described above. Any reference to memory, database, or other medium used in embodiments provided herein may include at least one of non-volatile and volatile memory. The nonvolatile Memory may include Read-Only Memory (ROM), magnetic tape, floppy disk, flash Memory, optical Memory, high density embedded nonvolatile Memory, resistive random access Memory (ReRAM), magnetic random access Memory (Magnetoresistive Random Access Memory, MRAM), ferroelectric Memory (Ferroelectric Random Access Memory, FRAM), phase change Memory (Phase Change Memory, PCM), graphene Memory, and the like. Volatile memory can include random access memory (Random Access Memory, RAM) or external cache memory, and the like. By way of illustration, and not limitation, RAM can be in the form of a variety of forms, such as static random access memory (Static Random Access Memory, SRAM) or dynamic random access memory (Dynamic Random Access Memory, DRAM), and the like. The databases referred to in the embodiments provided herein may include at least one of a relational database and a non-relational database. The non-relational database may include, but is not limited to, a blockchain-based distributed database, and the like. The processor referred to in the embodiments provided in the present application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic unit, a data processing logic unit based on quantum computing, or the like, but is not limited thereto.
The technical features of the above embodiments may be arbitrarily combined, and all possible combinations of the technical features in the above embodiments are not described for brevity of description, however, as long as there is no contradiction between the combinations of the technical features, they should be considered as the scope of the description.
The foregoing examples illustrate only a few embodiments of the application and are described in detail herein without thereby limiting the scope of the application. It should be noted that it will be apparent to those skilled in the art that several variations and modifications can be made without departing from the spirit of the application, which are all within the scope of the application. Accordingly, the scope of the application should be assessed as that of the appended claims.
Claims (11)
1. A method for expanding a financial database, applied to a metadata server, the method comprising:
responding to a capacity expansion instruction of a financial database, adding at least one capacity expansion storage node according to current configuration information sent by a front-end control console, and determining capacity expansion storage fragments matched with the capacity expansion storage nodes to obtain corresponding route change information; the capacity expansion storage partition comprises an original storage node and at least one capacity expansion storage node;
Updating the local routing information according to the routing change information;
under the condition that the local routing information updating is completed, sending a routing information updating instruction to each proxy node so as to enable each proxy node to suspend original writing service and update routing information;
and sending a service recovery instruction to the target agent node which has completed updating in the agent nodes, so that the target agent node executes corresponding read service according to the updated routing information and executes corresponding write service according to the updated routing information.
2. The method of claim 1, wherein the adding at least one expansion storage node and determining expansion storage fragments that the expansion storage node matches according to current configuration information sent by a front-end console comprises:
determining the newly increased number of the capacity expansion storage nodes according to the current configuration information;
and matching the capacity-expansion storage nodes with corresponding storage fragments according to a preset rule according to the newly-increased number of the capacity-expansion storage nodes and the arrangement sequence of the storage fragments to obtain the capacity-expansion storage fragments.
3. The method of claim 2, wherein the target proxy node performs a corresponding read service according to the updated routing information, comprising:
When the read request data is character string type data, the target agent node determines a target storage fragment according to the read request; forwarding the read request to all storage nodes in the capacity-expansion storage partition under the condition that the target storage partition is the capacity-expansion storage partition; and sending the target value to the metadata server under the condition that the target value exists in the returned result of any storage node.
4. The method of claim 2, wherein the target proxy node performs a corresponding read service according to the updated routing information, further comprising:
when the read request data is set type data, the target proxy node determines a target storage fragment according to the read request; when the target storage fragment is the capacity-expansion storage fragment, forwarding the read request to all storage nodes in the capacity-expansion storage fragment simultaneously; and receiving and combining the returned results of the storage nodes.
5. The method of claim 2, wherein the performing the corresponding write service based on the updated routing information comprises:
when the writing request data is character string type data, the target agent node determines a target storage fragment according to the writing request; and executing the writing service corresponding to the parameter information according to the parameter information in the writing request under the condition that the target storage fragment is the capacity expansion storage fragment.
6. The method of claim 2, wherein the performing the corresponding write service based on the updated routing information further comprises:
when the write request data is set type data, the target proxy node determines a target storage fragment according to the write request; and under the condition that the target storage fragment is the capacity-expansion storage fragment, inquiring whether a storage node which stores the target value in the writing request exists in the capacity-expansion storage fragment, and executing the writing service corresponding to the target set type according to the inquiring result.
7. The method of any of claims 1 to 6, wherein each proxy node pauses an original write service and updates routing information, comprising:
each proxy node responds to the routing information updating instruction and pauses the original writing service; acquiring the route change information from a metadata server, and updating the route information of the proxy node according to the route change information; sending updated feedback information to the metadata server under the condition that the routing information of the proxy node is updated;
before sending the service restoration instruction to the target agent node which has completed updating in the agent nodes, the method further comprises:
Receiving and counting the updated feedback information sent by the proxy node;
and identifying the agent node which completes updating within a preset time window as the target agent node according to the statistical result.
8. A financial database expansion device, the device comprising:
the node newly-added module is used for responding to the capacity-expanding instruction of the financial database, newly adding at least one capacity-expanding storage node according to the current configuration information sent by the front-end control console, and determining the capacity-expanding storage fragments matched with the capacity-expanding storage node to obtain corresponding route change information; the capacity expansion storage partition comprises an original storage node and at least one capacity expansion storage node;
the route updating module is used for updating the local route information according to the route change information;
the instruction sending module is used for sending routing information updating instructions to each proxy node under the condition that the local routing information updating is completed, so that each proxy node pauses the original writing service and updates the routing information;
and the service recovery module is used for sending a service recovery instruction to the target agent node which has completed updating in the agent nodes so that the target agent node executes corresponding read service according to the updated route information and executes corresponding write service according to the updated route information.
9. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that the processor implements the steps of the method of any of claims 1 to 7 when the computer program is executed.
10. A computer readable storage medium, on which a computer program is stored, characterized in that the computer program, when being executed by a processor, implements the steps of the method of any of claims 1 to 7.
11. A computer program product comprising a computer program, characterized in that the computer program, when being executed by a processor, implements the steps of the method of any of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311057220.3A CN117076470A (en) | 2023-08-22 | 2023-08-22 | Capacity expansion method, device, equipment, medium and program product for financial database |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311057220.3A CN117076470A (en) | 2023-08-22 | 2023-08-22 | Capacity expansion method, device, equipment, medium and program product for financial database |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117076470A true CN117076470A (en) | 2023-11-17 |
Family
ID=88711168
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311057220.3A Pending CN117076470A (en) | 2023-08-22 | 2023-08-22 | Capacity expansion method, device, equipment, medium and program product for financial database |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117076470A (en) |
-
2023
- 2023-08-22 CN CN202311057220.3A patent/CN117076470A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US6205451B1 (en) | Method and apparatus for incremental refresh of summary tables in a database system | |
| US11442961B2 (en) | Active transaction list synchronization method and apparatus | |
| CN112363979B (en) | Distributed index method and system based on graph database | |
| US11036720B2 (en) | Blockchain-based hierarchical data storage | |
| CN111309720A (en) | Time sequence data storage method, time sequence data reading method, time sequence data storage device, time sequence data reading device, electronic equipment and storage medium | |
| US11288247B2 (en) | Blockchain based hierarchical data storage | |
| CN110633378A (en) | Graph database construction method supporting super-large scale relational network | |
| CN113760847A (en) | Log data processing method, device, equipment and storage medium | |
| US20200242104A1 (en) | Method and system for data handling | |
| Li et al. | Accurate Counting Bloom Filters for Large‐Scale Data Processing | |
| CN112015819A (en) | Data updating method, device, equipment and medium for distributed graph database | |
| CN116303789A (en) | Parallel synchronization method and device for multi-fragment multi-copy database and readable medium | |
| US20200242132A1 (en) | Method and system for data handling | |
| CN117076470A (en) | Capacity expansion method, device, equipment, medium and program product for financial database | |
| US11372832B1 (en) | Efficient hashing of data objects | |
| US11163642B2 (en) | Methods, devices and computer readable medium for managing a redundant array of independent disks | |
| CN111191082B (en) | Data management method, device, computer equipment and storage medium | |
| CN114416689A (en) | Data migration method and device, computer equipment and storage medium | |
| CN111708626A (en) | Data access method and device, computer equipment and storage medium | |
| CN117131023B (en) | Data table processing method, device, computer equipment and readable storage medium | |
| CN113051274B (en) | Mass tag storage system and method | |
| CN116257531B (en) | Database space recovery method | |
| CN117075951A (en) | Program maintenance method, program maintenance device, computer equipment and storage medium | |
| CN117194729A (en) | Power data storage method, apparatus, device, storage medium, and program product | |
| CN117290354A (en) | Data processing method, device, computer equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |