CN111695018A - Data processing method and device, distributed network system and computer equipment - Google Patents
Data processing method and device, distributed network system and computer equipment Download PDFInfo
- Publication number
- CN111695018A CN111695018A CN201910189686.6A CN201910189686A CN111695018A CN 111695018 A CN111695018 A CN 111695018A CN 201910189686 A CN201910189686 A CN 201910189686A CN 111695018 A CN111695018 A CN 111695018A
- Authority
- CN
- China
- Prior art keywords
- node
- page
- access request
- nodes
- designated
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/95—Retrieval from the web
- G06F16/953—Querying, e.g. by the use of web search engines
Abstract
The invention discloses a data processing method and device, a distributed network system and computer equipment. Wherein, the method comprises the following steps: a designated node in the distributed network receives an access request of a page; the appointed node determines a sender of the access request; and when the node corresponding to the sender is consistent with the designated node, the designated node forwards the access request to other nodes in the distributed network. The invention solves the technical problems that in the related art, due to the fact that partial nodes are down in a distributed system, the system cannot immediately open the data managed by the down nodes for a user to access, and the system availability is reduced.
Description
Technical Field
The present invention relates to the field of data processing technologies, and in particular, to a data processing method and apparatus, a distributed network system, and a computer device.
Background
In a distributed storage database system, a plurality of distributed nodes are involved, cache is shared among the nodes, and meanwhile, each node stores state information of pages managed by the node. However, some nodes are down when sending page access requests, and cannot immediately recover data, which results in that page information is lost after being down, for example, some pages are modified by and managed by the down nodes, and the information of these pages is lost after being down, at this time, the system cannot immediately recover the page information managed by the down nodes, so that it is impossible to accurately know which pages are being modified and need to be recovered before being down, and finally, the system cannot immediately open access to the data managed by all the down nodes, and the system availability is reduced.
In view of the above problems, no effective solution has been proposed.
Disclosure of Invention
Embodiments of the present invention provide a data processing method and apparatus, a distributed network system, and a computer device, so as to at least solve the technical problems in the related art that, due to a shutdown occurring in some nodes in a distributed system, the system cannot immediately open data managed by the shutdown node for a user to access, and the availability of the system is reduced.
According to an aspect of an embodiment of the present invention, there is provided a data processing method including: a designated node in the distributed network receives an access request of a page; the designated node determines a sender of the access request; and when the node corresponding to the sender is consistent with the designated node, the designated node forwards the access request to other nodes in the distributed network.
According to another aspect of the embodiments of the present invention, there is also provided a data processing method, including: a designated node in the distributed network receives an access request of a page; the designated node determines a sender of the access request; and when the node corresponding to the sender is consistent with the designated node, the designated node informs other nodes in the distributed network of the current state of the page.
According to another aspect of the embodiments of the present invention, there is also provided a data processing method, including: a designated node in the distributed network receives an access request of a page; the designated node forwards the access request to other nodes in the distributed network; the other node determines the type of the access request; and when the type is an exclusive lock request, the other nodes record the state information of the page, wherein the state information is used for indicating that the page is in a locked state.
According to another aspect of the embodiments of the present invention, there is also provided a distributed network system, including: a plurality of distributed nodes; wherein, a designated node in the plurality of distributed nodes is used for receiving an access request of a page; determining a sender of the access request; and when the node corresponding to the sender is consistent with the designated node, forwarding the access request to other nodes in the distributed network.
According to another aspect of the embodiments of the present invention, there is also provided a data processing apparatus, which is applied to a designated node of a distributed network, the apparatus including: the receiving module is used for receiving an access request of a page; a determining module for determining a sender of the access request; and the sending module is used for forwarding the access request to other nodes in the distributed network when the node corresponding to the sender is consistent with the designated node.
According to another aspect of the embodiments of the present invention, there is also provided a computer device, including: a processor; and a memory coupled to the processor for providing instructions to the processor for processing the following processing steps: receiving an access request of a page; determining a sender of the access request; and when the node corresponding to the sender is consistent with the computer equipment, forwarding the access request to other nodes in the distributed network.
In the embodiment of the invention, the appointed node in the distributed network is adopted to receive the access request of the page, the appointed node determines the sender of the access request, and when the node corresponding to the sender is consistent with the appointed node, the appointed node forwards the access request to other nodes in the distributed network. In this embodiment, when there is an access request, it may be determined whether a request sender is consistent with a designated node, if so, the access request may be forwarded to other nodes in the distributed network, and the state information of the page corresponding to the access request is cached by other nodes, thereby implementing multiple storage of data, avoiding that the page information is lost after downtime when the designated node is a management node, and caching the state information of the page by other nodes, so that the state information of the page of the access request may be retrieved by other nodes after downtime occurs on the designated node, and data processing is completed, so that the access delay occurring during downtime or the frequency of loss of the access request may be reduced, the time during data recovery in which data is inaccessible is greatly reduced, the availability of the system is improved, and further solving the problem that some nodes in the distributed system in the related art are down, the system cannot immediately open the data managed by the downtime node for the user to access, and the usability of the system is reduced.
Drawings
The accompanying drawings, which are included to provide a further understanding of the invention and are incorporated in and constitute a part of this application, illustrate embodiment(s) of the invention and together with the description serve to explain the invention without limiting the invention. In the drawings:
fig. 1 shows a hardware configuration block diagram of a computer terminal for implementing a data processing method;
FIG. 2 shows a schematic diagram of a network terminal for a data processing method;
FIG. 3 is a flow chart of a data processing method according to a first embodiment of the invention;
FIG. 4 is a flow chart one of another alternative data processing method according to an embodiment of the present invention;
FIG. 5 is a flow chart diagram two of another alternative data processing method according to an embodiment of the present invention;
FIG. 6 is a schematic diagram of a distributed network according to an embodiment of the present invention;
FIG. 7 is a schematic diagram of an alternative data processing apparatus according to an embodiment of the present invention;
fig. 8 is a block diagram of a computer terminal according to an embodiment of the present invention.
Detailed Description
In order to make the technical solutions of the present invention better understood, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
It should be noted that the terms "first," "second," and the like in the description and claims of the present invention and in the drawings described above are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used is interchangeable under appropriate circumstances such that the embodiments of the invention described herein are capable of operation in sequences other than those illustrated or described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
Example 1
There is also provided, in accordance with an embodiment of the present invention, an embodiment of a data processing method, it should be noted that the steps illustrated in the flowchart of the accompanying drawings may be performed in a computer system such as a set of computer executable instructions, and that, although a logical order is illustrated in the flowchart, in some cases, the steps illustrated or described may be performed in an order different than that herein.
The data processing method provided by the first embodiment of the present application may be executed in a mobile terminal, a computer terminal, or a similar computing device. Fig. 1 shows a hardware configuration block diagram of a computer terminal (or mobile device) for implementing a data processing method. As shown in fig. 1, the computer terminal 10 (or mobile device 10) may include one or more (shown as 102a, 102b, … …, 102 n) processors 102 (the processors 102 may include, but are not limited to, a processing device such as a microprocessor MCU or a programmable logic device FPGA, etc.), a memory 104 for storing data, and a transmission device 106 for communication functions. Besides, the method can also comprise the following steps: a display, an input/output interface (i.e., an I/O interface), a Universal Serial Bus (USB) port (which may be included as one of the ports of the I/O interface), a network interface, a power source, and/or a camera. It will be understood by those skilled in the art that the structure shown in fig. 1 is only an illustration and is not intended to limit the structure of the electronic device. For example, the computer terminal 10 may also include more or fewer components than shown in FIG. 1, or have a different configuration than shown in FIG. 1.
It should be noted that the one or more processors 102 and/or other data processing circuitry described above may be referred to generally herein as "data processing circuitry". The data processing circuitry may be embodied in whole or in part in software, hardware, firmware, or any combination thereof. Further, the data processing circuit may be a single stand-alone processing module, or incorporated in whole or in part into any of the other elements in the computer terminal 10 (or mobile device). As referred to in the embodiments of the application, the data processing circuit acts as a processor control (e.g. selection of a variable resistance termination path connected to the interface).
The memory 104 may be used to store software programs and modules of application software, such as program instructions/data storage devices corresponding to the data processing method in the embodiment of the present invention, and the processor 102 executes various functional applications and data processing by executing the software programs and modules stored in the memory 104, that is, implementing the data processing method described above. The memory 104 may include high speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include memory located remotely from the processor 102, which may be connected to the computer terminal 10 via a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
The transmission device 106 is used for receiving or transmitting data via a network. Specific examples of the network described above may include a wireless network provided by a communication provider of the computer terminal 10. In one example, the transmission device 106 includes a Network adapter (NIC) that can be connected to other Network devices through a base station to communicate with the internet. In one example, the transmission device 106 can be a Radio Frequency (RF) module, which is used to communicate with the internet in a wireless manner.
The display may be, for example, a touch screen type Liquid Crystal Display (LCD) that may enable a user to interact with a user interface of the computer terminal 10 (or mobile device).
Fig. 1 shows a block diagram of a hardware structure, which may be taken as an exemplary block diagram of the computer terminal 10 (or the mobile device) and also taken as an exemplary block diagram of the server, and in an alternative embodiment, fig. 2 shows a schematic diagram of a data processing method network terminal, and as shown in fig. 2, the computer terminal 10 (or the mobile device) may be connected or electronically connected to one or more servers (e.g., a security server, a resource server, a game server, etc.) via a data network. In an alternative embodiment, the computer terminal 10 (or mobile device) may be any mobile computing device or the like. The data network connection may be a local area network connection, a wide area network connection, an internet connection, or other type of data network connection. The computer terminal 10 (or mobile device) may execute to connect to a network service executed by a server (e.g., a secure server) or a group of servers. A web server is a network-based user service such as social networking, cloud resources, email, online payment, or other online applications.
Under the above operating environment, the present application provides a data processing method as shown in fig. 3. Fig. 3 is a flowchart of a data processing method according to a first embodiment of the present invention. As shown in fig. 3, the method includes:
step S302, a designated node in the distributed network receives an access request of a page;
step S304, the appointed node determines the sender of the access request;
and step S306, when the node corresponding to the sender is consistent with the designated node, the designated node forwards the access request to other nodes in the distributed network.
Through the steps, the appointed node in the distributed network can be adopted to receive the access request of the page, the appointed node determines the sender of the access request, and when the node corresponding to the sender is consistent with the appointed node, the appointed node is utilized to forward the access request to other nodes in the distributed network. In this embodiment, when there is an access request, it may be determined whether a request sender is consistent with a designated node, if so, the access request may be forwarded to other nodes in the distributed network, and the state information of the page corresponding to the access request is cached by other nodes, thereby implementing multiple storage of data, avoiding that the page information is lost after downtime when the designated node is a management node, and caching the state information of the page by other nodes, so that the state information of the page of the access request may be retrieved by other nodes after downtime occurs on the designated node, and data processing is completed, so that the access delay occurring during downtime or the frequency of loss of the access request may be reduced, the time during data recovery in which data is inaccessible is greatly reduced, the availability of the system is improved, and further solving the problem that some nodes in the distributed system in the related art are down, the system cannot immediately open the data managed by the downtime node for the user to access, and the usability of the system is reduced.
The embodiment of the application can be applied to a distributed system, especially a distributed shared database system, for example, a plurality of nodes in a distributed network are managed. According to the embodiment of the application, the distributed system or the distributed network is managed from the perspective of the designated node, and data processing of the down of the node is achieved. When a node goes down, other nodes are selected for data recovery, in the recovery process, all the page information modified by the down node can be immediately and completely recovered from all the nodes, at the moment, the modified pages can be locked, and the reading and writing of all the data can be recovered.
The present application will be described below with reference to the respective steps.
Step S302, a designated node in the distributed network receives an access request of a page.
In the embodiment of the present invention, the distributed network may be a network including a plurality of nodes, the network structure includes nodes distributed at different locations and having a plurality of terminals connected to each other, any node in the network is connected to at least two lines, and when any line fails, communication may be completed via another link.
In the embodiment of the invention, the network comprises a plurality of nodes for sharing cache data, and other nodes are managed through one or more designated nodes, wherein an information table is recorded in the designated nodes, and the information table comprises each managed node and cache data or cache data pages in each node. If a node (node 1) needs to access a data page, an access request needs to be sent to a designated node (node2) corresponding to the data page, the designated node (node2) checks an information table stored by itself, determines which node cache currently has the data page, sends the access request to the node (node 3), and the node (node 3) directly sends the data page to the initial node (namely, node 1).
In some embodiments of the present application, each node maintains state information and page data for the pages it manages.
The designated Node in the embodiment of the present invention may be defined as a modified Node and a management Node, and the number of other nodes managed by the designated Node is not limited in the embodiment of the present invention, for example, the defined Node2 manages other three nodes Node1, Node3, and Node 4.
When a page data request is specifically processed, a page access request is received through the designated node, and since a distributed network may include a plurality of designated nodes, each of which manages a plurality of nodes, when sending the access request, it needs to manage its own designated node to send the access request to obtain corresponding page data.
The access request of the page may include, but is not limited to: for content requests of web page videos, articles, voices, songs and the like, a distributed node may cache a plurality of page data.
In step S304, the designated node determines the sender of the access request.
After the designated node receives the access request of the page, it may first determine which node sends the access request, and in an alternative embodiment, the determining, by the designated node, the sender of the access request includes: the appointed node acquires the page identifier of the page from the access request; and the designated node determines a node identifier corresponding to the page based on the page identifier, and takes the node corresponding to the node identifier as a sender.
The page identifier of the page may include, but is not limited to: page links (e.g., URLs), identifying numbers and/or letters on the page, page rank numbers, etc. The node to which the access request belongs can be determined through the page identification, and then the sender of the access request is determined.
In another optional embodiment of the present invention, the determining, by the designated node, the node identifier corresponding to the page based on the page identifier includes: carrying out hash operation on the page identification by the designated node to obtain a hash value; and determining the node identification corresponding to the page based on the hash value.
Namely, in some embodiments of the present invention, hash operation may be performed on the page identifier, and the node identifier to which the page identifier belongs may be determined by the hash value. For example, if the page identifier is the page number of the page, hash operation is performed on the page number, and the node identifier corresponding to the page is determined according to the hash value. Certainly, since the access request for sending the page may be the designated Node, or may be other nodes managed by the designated Node, Node identifiers of different nodes are different, if the access request for sending the page is the designated Node, for example, the designated Node2, the Node identifier may be determined according to a Hash value by making a Hash of the page number.
The node identification may include: node number, node code, etc., and the node identifier may be a designated node number or other node numbers. The designated node number corresponding to the access request of the page can be determined through the hash value.
In some embodiments of the present invention, if a node needs to access a data page, an access request needs to be sent to a designated node corresponding to the data page. In the first case, other nodes send access requests to the designated nodes, that is, other nodes need to acquire page data; and secondly, the designated node sends an access request to the designated node, namely the designated node also needs to acquire corresponding page data. For the first case, because other nodes send access requests to their designated nodes, when they send access requests, they will record the content of the access requests themselves, and record the content of the access requests through the designated nodes or other nodes, and at the same time, the nodes visited by the pages of each access request will record the state information of the pages, so that even if the designated nodes are down or otherwise have faults, the page states and the contents of the accessed pages of the pages corresponding to the access requests can be acquired and recovered from other nodes. For the second case, because the node requesting access is the self, and is not cached by other nodes, the situation that the page data cannot be accessed may occur during downtime, and only after waiting for the Redo log of the downtime node to be scanned once, the set of the modified pages is acquired, and then the other page information sets acquired from other nodes, the union of the two sets is obtained, and after all the pages in the union are locked, the data originally managed by the downtime node can be accessed (the access to the locked pages is recovered and completed by the blocked specified pages).
And step S306, when the node corresponding to the sender is consistent with the designated node, the designated node forwards the access request to other nodes in the distributed network.
That is, in the embodiment of the present invention, after the designated node receives the access request from the sender, if it is determined that the sender is the designated node itself, the access request may be forwarded to other nodes in the distributed network, and then the page state of the page corresponding to the access request is cached by the other nodes, so as to perform notification (record) processing.
Optionally, in the embodiment of the present invention, if the designated node sends the access request to the other nodes, the other nodes may record state information of the page requested by the access request (record that the page is exclusively locked), so that when the designated node goes down, the page state of the page requested by the access request is acquired from the other nodes.
In another optional embodiment of the present invention, before the designated node forwards the access request to other nodes in the distributed network, the method further includes: the designated node determines a target node number based on the node identification of the designated node and the number of nodes in the distributed network; and taking the node corresponding to the target node number as the other node.
When other nodes are selected, the node identification of the designated node and the number of nodes in the distributed network need to be referred to, wherein the position of the designated node can be determined through the node identification of the designated node, and the nodes around the designated node can be inquired, the surrounding nodes can be preferably selected to serve as the other nodes, and the fact that several other nodes are required to cache the access request can be determined through the number of nodes in the distributed network. For example, the node number of the designated node is determined, and then the access request is forwarded to the ((designated node number + 1)% total node number) th node.
After receiving the access request, the other nodes may record the state information of the page requested by the access request. In an optional embodiment of the present invention, the method further includes: other nodes determine the type of the access request; when the type is the exclusive lock request, the state information of the page is recorded, and the state information is used for indicating that the page is in a locked state. That is, if the type of the access request is an exclusive lock request, it is necessary to record the state information of the page, and the state information indicates that the page is in a locked state.
When the page lock request sent to the master node of the page (i.e. the node managing the page) comes from the node (i.e. the requester is the master node), the request is sent to another node (i.e. the other node) for storage at the same time, so that two copies of the lock information of the page can be reserved. Thus, all pages have information held by at least two nodes: for example, when the master node of page 0 is node1, if node2 requests this page, then there will be information for the page at both node2 and node1 (who holds what lock the page has); if node1 requests page 0, node1 will forward the request message to another node (e.g., node3) so that node3 also has the locking information (avoiding only node1 having information, so that when any node hangs up, the locking information can always be re-established from the other nodes.
Optionally, after the designated node forwards the access request to other nodes in the distributed network, the method further includes: after the specified node is down, acquiring the state information of the page from other nodes; the designated node restores the page state of the page based on the state information of the page.
That is, when the designated node goes down, one of the remaining nodes (that is, the other nodes) may be selected to perform data recovery, in the recovery process, the state information of the page may be acquired from the other nodes, and then the designated node recovers the page state of the page based on the state information of the page, if the state information of the page indicates that the page is in a locked state, in this way, during the recovery, the page may be unlocked and the page data of the page may be continuously acquired.
In the embodiment of the invention, all the page information modified by the down node can be immediately and completely recovered from all the nodes. Similarly, the designated node may also recover the page state of the page through the page information according to the state information of the page acquired by the other node.
Another optional feature of the method further comprises: when the type is not an exclusive lock request and the node corresponding to the sender is consistent with the designated node, determining the node corresponding to the page; the appointed node sends an access request to a node corresponding to the page; and the designated node receives page data sent by the node corresponding to the page, wherein the page data is the page data corresponding to the page.
That is, if it is determined that the type of the access request is not the exclusive lock request, the page corresponding to the access request may be directly accessed, so that the designated node may send the access request to the node corresponding to the page, and then receive page data sent by the node corresponding to the page, where the page data is page data corresponding to the page. For example, the designated Node (Node2) determines which Node's cache currently holds the page, and sends a request to the Node (e.g., Node1), and the Node receives the request and directly sends the page to the first read requesting Node (e.g., Node 3).
Through the embodiment, data processing can be performed in the distributed network, in the processing process, if the access request for sending the page is the designated node (namely, the management node) itself, the access request can be sent to one other node, and the other node records the state information of the page corresponding to the access request (namely, records that the page is added with an exclusive lock), so that even if the designated node goes down, the state information of the page can be obtained through the other node, the page state of the page is recovered based on the state information of the page, and then the page data sent by the node corresponding to the page is received, so that the data inaccessible time in the data recovery process can be greatly reduced, and the availability of the system is improved.
The invention is illustrated below by means of a further alternative embodiment.
Fig. 4 is a first flowchart of another alternative data processing method according to an embodiment of the present invention, as shown in fig. 4, the method includes:
step S402, a designated node in the distributed network receives an access request of a page;
step S404, appointing a node to determine a sender of the access request;
step S406, when the node corresponding to the sender is consistent with the designated node, the designated node notifies other nodes in the distributed network of the current state of the page.
Through the steps, the appointed node in the distributed network is adopted to receive the access request of the page, the appointed node determines the sender of the access request, and when the node corresponding to the sender is consistent with the appointed node, the appointed node informs other nodes in the distributed network of the current state of the page. In this embodiment, in the distributed network, if the sender is the designated node itself, the current state of the page may be recorded by other nodes, so that even if the designated node goes down, the state information of the page may be retrieved by other nodes to complete data processing, thereby reducing the access delay or the frequency of losing access requests when the designated node goes down, greatly reducing the time during which data is inaccessible in the data recovery process, and improving the availability of the system.
Optionally, the notifying, by the designated node, the current state of the page to other nodes in the distributed network includes: the designated node sends an access request to other nodes, and the access request is used for determining the state of the page.
That is, a state access request can be sent by a designated node to determine the state of the page where the access request for sending the page is located, and optionally, the state can be an exclusive lock or a non-exclusive lock. The state can be determined directly by the access request, for example, when the type is an exclusive lock request, state information of the page is recorded, and the state information is used for indicating that the page is in a locked state; when the type is not an exclusive lock request and the node corresponding to the sender is consistent with the designated node, determining the node corresponding to the page; the appointed node sends an access request to a node corresponding to the page; and the designated node receives page data sent by the node corresponding to the page, wherein the page data is the page data corresponding to the page.
The invention is illustrated below by means of a further alternative embodiment.
Fig. 5 is a second flowchart of another alternative data processing method according to an embodiment of the present invention, as shown in fig. 5, the method includes:
step S502, a designated node in the distributed network receives an access request of a page;
step S504, the appointed node transmits the access request to other nodes in the distributed network;
step S506, other nodes determine the type of the access request;
in step S508, when the type is the exclusive lock request, the other nodes record the state information of the page, where the state information is used to indicate that the page is in the locked state.
Through the steps, the designated node in the distributed network receives the access request of the page, the designated node forwards the access request to other nodes in the distributed network, the other nodes determine the type of the access request, and when the type of the other nodes is the exclusive lock request, the other nodes record the state information of the page, wherein the state information is used for indicating that the page is in the locking state. In this embodiment, the access request may be forwarded to other nodes in the distributed network through the designated node, and the state information of the page of the access request is recorded by using other nodes, so as to implement multiple storage of data, avoid that the page information is lost after downtime when the designated node is a management node, retrieve the state information of the page sending the access request through other nodes, and complete data processing, thereby reducing access delay or access request loss frequency occurring during downtime, greatly reducing data inaccessibility time in the data recovery process, improving system availability, and further solving the technical problems that in the related art, due to downtime of some nodes in the distributed system, the system cannot immediately open the data managed by the downtime node for a user to access, and the system availability is reduced.
It should be noted that, for simplicity of description, the above-mentioned method embodiments are described as a series of acts or combination of acts, but those skilled in the art will recognize that the present invention is not limited by the order of acts, as some steps may occur in other orders or concurrently in accordance with the invention. Further, those skilled in the art should also appreciate that the embodiments described in the specification are preferred embodiments and that the acts and modules referred to are not necessarily required by the invention.
Through the above description of the embodiments, those skilled in the art can clearly understand that the data processing method according to the above embodiments can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware, but the former is a better implementation mode in many cases. Based on such understanding, the technical solutions of the present invention may be embodied in the form of a software product, which is stored in a storage medium (e.g., ROM/RAM, magnetic disk, optical disk) and includes instructions for enabling a terminal device (e.g., a mobile phone, a computer, a server, or a network device) to execute the method according to the embodiments of the present invention.
Example 2
According to an embodiment of the present invention, there is also provided a distributed network for implementing the data processing method, and fig. 6 is a schematic diagram of a distributed network according to an embodiment of the present invention, as shown in fig. 6, the distributed network includes: a plurality of distributed nodes, which may be designated node 61, distributed node 62, distributed node 63, distributed node 64; the system comprises a plurality of distributed nodes, a designated node and a plurality of distributed nodes, wherein the designated node in the plurality of distributed nodes is used for receiving an access request of a page; determining a sender of the access request; and when the node corresponding to the sender is consistent with the designated node, forwarding the access request to other nodes in the distributed network.
The distributed network can receive the access request of the page through a designated node in the plurality of distributed nodes, determine the sender of the access request, and forward the access request to other nodes in the distributed network when the node corresponding to the sender is consistent with the designated node. In this embodiment, when there is an access request, it may be determined whether a request sender is consistent with a designated node, if so, the access request may be forwarded to other nodes in the distributed network, the state information of the page corresponding to the access request is cached by the other nodes, so as to avoid that the page information is lost after downtime when the designated node is a management node, and the state information of the page is cached by the other nodes, so that the state information of the page of the access request may be retrieved by the other nodes after downtime occurs at the designated node, and data processing is completed, so that the access delay occurring during downtime or the frequency of loss of the access request may be reduced, the data inaccessible time during data recovery is greatly reduced, the availability of the system is improved, and further, the problem that a part of nodes in the distributed system are down in the related art is solved, the system cannot immediately open the data managed by the downtime node for the user to access, and the usability of the system is reduced.
Example 3
According to an embodiment of the present invention, there is also provided a data processing apparatus, and fig. 7 is a schematic diagram of an alternative data processing apparatus according to an embodiment of the present invention, which is applied in a designated node of a distributed network, and as shown in fig. 7, the apparatus may include: a receiving module 71, a determining module 73, a sending module 75, wherein,
a receiving module 71, configured to receive an access request of a page;
a determining module 73, configured to determine a sender of the access request;
a sending module 75, configured to forward the access request to other nodes in the distributed network when the node corresponding to the sender is consistent with the designated node.
The data processing apparatus may receive an access request of a page through the receiving module 71, determine a sender of the access request through the determining module 73, and forward the access request to another node in the distributed network through the sending module 75 when a node corresponding to the sender is consistent with the designated node. In this embodiment, when there is an access request, it may be determined whether a request sender is consistent with a designated node, if so, the access request may be forwarded to other nodes in the distributed network, and the state information of the page corresponding to the access request is cached by other nodes, thereby implementing multiple storage of data, avoiding that the page information is lost after downtime when the designated node is a management node, and caching the state information of the page by other nodes, so that the state information of the page of the access request may be retrieved by other nodes after downtime occurs on the designated node, and data processing is completed, so that the access delay occurring during downtime or the frequency of loss of the access request may be reduced, the time during data recovery in which data is inaccessible is greatly reduced, the availability of the system is improved, and further solving the problem that some nodes in the distributed system in the related art are down, the system cannot immediately open the data managed by the downtime node for the user to access, and the usability of the system is reduced.
The data processing device may further include a processor and a memory, and the receiving module 71, the determining module 73, the sending module 75, and the like are stored in the memory as program units, and the processor executes the program units stored in the memory to implement corresponding functions.
The processor comprises a kernel, and the kernel calls a corresponding program unit from the memory. The kernel can set one or more than one, and the access request is forwarded to other nodes in the distributed network when the node corresponding to the sender is consistent with the designated node by adjusting kernel parameters.
The memory may include volatile memory in a computer readable medium, Random Access Memory (RAM) and/or nonvolatile memory such as Read Only Memory (ROM) or flash memory (flash RAM), and the memory includes at least one memory chip.
It should be noted here that the receiving module 71, the determining module 73, and the sending module 75 correspond to steps S304 to S306 in embodiment 1, and the three modules are the same as the corresponding steps in the implementation example and application scenario, but are not limited to the disclosure in the first embodiment. It should be noted that the modules described above as part of the apparatus may be run in the computer terminal 10 provided in the first embodiment.
Example 4
Embodiments of the present invention may provide a computer device, which may be any one of a group of computer devices, located in a distributed network. Optionally, in this embodiment, the computer device may also be replaced with a device such as a mobile terminal.
Optionally, in this embodiment, the computer device may be located in at least one network device of a plurality of network devices of a distributed computer network.
Optionally, the computer device comprises: a processor; and a memory coupled to the processor for providing instructions to the processor for processing the following processing steps: receiving an access request of a page; determining a sender of the access request; and when the node corresponding to the sender is consistent with the computer equipment, forwarding the access request to other nodes in the distributed network.
In this embodiment, the computer device described above may further execute program code for the following steps in the data processing method: acquiring a page identifier of a page from the access request; and determining a node identifier corresponding to the page based on the page identifier, and taking the node corresponding to the node identifier as a sender.
In this embodiment, the computer device described above may further execute program code for the following steps in the data processing method: carrying out hash operation on the page identification to obtain a hash value; and determining the node identification corresponding to the page based on the hash value.
In this embodiment, the computer device described above may further execute program code for the following steps in the data processing method: determining a target node number based on the node identification of the designated node and the number of nodes in the distributed network; and taking the node corresponding to the target node number as the other node.
In this embodiment, the computer device described above may further execute program code for the following steps in the data processing method: other nodes determine the type of the access request; when the type is the exclusive lock request, the state information of the page is recorded, and the state information is used for indicating that the page is in a locked state.
In this embodiment, the computer device described above may further execute program code for the following steps in the data processing method: after the access request is forwarded to other nodes in the distributed network, the designated node acquires the state information of the page from other nodes after the node is down; the designated node restores the page state of the page based on the state information of the page.
In this embodiment, the computer device described above may further execute program code for the following steps in the data processing method: when the type is not an exclusive lock request and the node corresponding to the sender is consistent with the designated node, determining the node corresponding to the page; the appointed node sends an access request to a node corresponding to the page; and the designated node receives page data sent by the node corresponding to the page, wherein the page data is the page data corresponding to the page.
Alternatively, fig. 8 is a block diagram of a computer terminal according to an embodiment of the present invention. As shown in fig. 8, the computer terminal a may include: one or more processors, memory, and network interfaces, input/output interfaces, keyboards, displays.
The memory may be configured to store software programs and modules, such as program instructions/modules corresponding to the data processing method and apparatus in the embodiments of the present invention, and the processor executes various functional applications and data processing by running the software programs and modules stored in the memory, so as to implement the data processing method. The memory may include high speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory may further include memory remotely located from the processor, and these remote memories may be connected to terminal a through a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
The processor can call the information and application program stored in the memory through the transmission device to execute the following steps: receiving an access request of a page; determining a sender of the access request; and when the node corresponding to the sender is consistent with the computer equipment, forwarding the access request to other nodes in the distributed network.
Optionally, the processor may further execute the program code of the following steps: acquiring a page identifier of a page from the access request; and determining a node identifier corresponding to the page based on the page identifier, and taking the node corresponding to the node identifier as a sender.
Optionally, the processor may further execute the program code of the following steps: carrying out hash operation on the page identification to obtain a hash value; and determining the node identification corresponding to the page based on the hash value.
Optionally, the processor may further execute the program code of the following steps: determining a target node number based on the node identification of the designated node and the number of nodes in the distributed network; and taking the node corresponding to the target node number as the other node.
Optionally, the processor may further execute the program code of the following steps: other nodes determine the type of the access request; when the type is the exclusive lock request, the state information of the page is recorded, and the state information is used for indicating that the page is in a locked state.
Optionally, the processor may further execute the program code of the following steps: after the access request is forwarded to other nodes in the distributed network, the designated node acquires the state information of the page from other nodes after the node is down; the designated node restores the page state of the page based on the state information of the page.
The embodiment of the invention provides a data processing scheme. The method comprises the steps that the designated nodes in the distributed network are adopted to receive the access requests of the pages, the designated nodes determine the sending party of the access requests, and when the nodes corresponding to the sending party are consistent with the designated nodes, the designated nodes forward the access requests to other nodes in the distributed network, so that the access delay or the access request loss frequency occurring in the downtime process is reduced, the data inaccessible time in the data recovery process is greatly reduced, and the usability of the system is improved.
It can be understood by those skilled in the art that the structure shown in fig. 8 is only an illustration, and the computer terminal may also be a terminal device such as a smart phone (e.g., an Android phone, an iOS phone, etc.), a tablet computer, a palmtop computer, a Mobile Internet Device (MID), a PAD, and the like. Fig. 8 does not limit the structure of the electronic device. For example, the computer terminal a may also include more or fewer components (e.g., network interfaces, display devices, etc.) than shown in fig. 8, or have a different configuration than shown in fig. 8.
Those skilled in the art will appreciate that all or part of the steps in the methods of the above embodiments may be implemented by a program instructing hardware associated with the terminal device, where the program may be stored in a computer-readable storage medium, and the storage medium may include: flash disks, Read-Only memories (ROMs), Random Access Memories (RAMs), magnetic or optical disks, and the like.
Example 5
The embodiment of the invention also provides a storage medium. Optionally, in this embodiment, the storage medium may be configured to store a program code executed by the data processing method provided in the first embodiment.
Optionally, in this embodiment, the storage medium may be located in any one of computer terminals in a computer terminal group in a computer network, or in any one of mobile terminals in a mobile terminal group.
Optionally, in this embodiment, the storage medium is configured to store program code for performing the following steps: receiving an access request of a page; determining a sender of the access request; and when the node corresponding to the sender is consistent with the computer equipment, forwarding the access request to other nodes in the distributed network.
The above-mentioned serial numbers of the embodiments of the present invention are merely for description and do not represent the merits of the embodiments.
In the above embodiments of the present invention, the descriptions of the respective embodiments have respective emphasis, and for parts that are not described in detail in a certain embodiment, reference may be made to related descriptions of other embodiments.
In the embodiments provided in the present application, it should be understood that the disclosed technology can be implemented in other ways. The above-described embodiments of the apparatus are merely illustrative, and for example, the division of the units is only one type of division of logical functions, and there may be other divisions when actually implemented, for example, a plurality of units or components may be combined or may be integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, units or modules, and may be in an electrical or other form.
The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.
In addition, functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.
The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a removable hard disk, a magnetic or optical disk, and other various media capable of storing program codes.
The foregoing is only a preferred embodiment of the present invention, and it should be noted that, for those skilled in the art, various modifications and decorations can be made without departing from the principle of the present invention, and these modifications and decorations should also be regarded as the protection scope of the present invention.
Claims (13)
1. A data processing method, comprising:
a designated node in the distributed network receives an access request of a page;
the designated node determines a sender of the access request;
and when the node corresponding to the sender is consistent with the designated node, the designated node forwards the access request to other nodes in the distributed network.
2. The method of claim 1, wherein the designated node determining the sender of the access request comprises:
the designated node acquires the page identifier of the page from the access request;
and the designated node determines a node identifier corresponding to the page based on the page identifier, and takes the node corresponding to the node identifier as the sender.
3. The method of claim 2, wherein the determining, by the designated node, the node identifier corresponding to the page based on the page identifier comprises:
the designated node performs hash operation on the page identifier to obtain a hash value; and determining the node identification corresponding to the page based on the hash value.
4. The method of claim 1, wherein before the designated node forwards the access request to other nodes in the distributed network, the method further comprises:
the designated node determines a target node number based on the node identification of the designated node and the number of nodes in the distributed network; and taking the node corresponding to the target node number as the other node.
5. The method according to any one of claims 1 to 4, further comprising:
the other node determines the type of the access request;
and when the type is an exclusive lock request, recording the state information of the page, wherein the state information is used for indicating that the page is in a locked state.
6. The method of claim 5, wherein after the designated node forwards the access request to other nodes in the distributed network, the method further comprises:
after the specified node is down, acquiring the state information of the page from other nodes; and the designated node restores the page state of the page based on the state information of the page.
7. The method of claim 5, further comprising:
when the type is not the exclusive lock request and the node corresponding to the sender is consistent with the designated node, determining the node corresponding to the page;
the designated node sends the access request to a node corresponding to the page;
and the designated node receives page data sent by the node corresponding to the page, wherein the page data is the page data corresponding to the page.
8. A data processing method, comprising:
a designated node in the distributed network receives an access request of a page;
the designated node determines a sender of the access request;
and when the node corresponding to the sender is consistent with the designated node, the designated node informs other nodes in the distributed network of the current state of the page.
9. The method of claim 8, wherein the designated node notifies other nodes in the distributed network of the current state of the page, comprising:
and the specified node sends the access request to the other nodes, and the access request is used for determining the state of the page.
10. A data processing method, comprising:
a designated node in the distributed network receives an access request of a page;
the designated node forwards the access request to other nodes in the distributed network;
the other node determines the type of the access request;
and when the type is an exclusive lock request, the other nodes record the state information of the page, wherein the state information is used for indicating that the page is in a locked state.
11. A distributed network system, comprising: a plurality of distributed nodes; wherein the content of the first and second substances,
a designated node of the plurality of distributed nodes for receiving an access request of a page; determining a sender of the access request; and when the node corresponding to the sender is consistent with the designated node, forwarding the access request to other nodes in the distributed network.
12. A data processing apparatus for use in a designated node of a distributed network, the apparatus comprising:
the receiving module is used for receiving an access request of a page;
a determining module for determining a sender of the access request;
and the sending module is used for forwarding the access request to other nodes in the distributed network when the node corresponding to the sender is consistent with the designated node.
13. A computer device located in a distributed network, comprising:
a processor; and
a memory coupled to the processor for providing instructions to the processor for processing the following processing steps:
receiving an access request of a page; determining a sender of the access request; and when the node corresponding to the sender is consistent with the computer equipment, forwarding the access request to other nodes in the distributed network.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910189686.6A CN111695018B (en) | 2019-03-13 | 2019-03-13 | Data processing method and device, distributed network system and computer equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910189686.6A CN111695018B (en) | 2019-03-13 | 2019-03-13 | Data processing method and device, distributed network system and computer equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111695018A true CN111695018A (en) | 2020-09-22 |
| CN111695018B CN111695018B (en) | 2023-05-30 |
Family
ID=72475646
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910189686.6A Active CN111695018B (en) | 2019-03-13 | 2019-03-13 | Data processing method and device, distributed network system and computer equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111695018B (en) |
Citations (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JPH06318950A (en) * | 1993-05-10 | 1994-11-15 | Pfu Ltd | Control system for tree structure network |
| US20020055989A1 (en) * | 2000-11-08 | 2002-05-09 | Stringer-Calvert David W.J. | Methods and apparatus for scalable, distributed management of virtual private networks |
| US20050237926A1 (en) * | 2004-04-22 | 2005-10-27 | Fan-Tieng Cheng | Method for providing fault-tolerant application cluster service |
| CN1719831A (en) * | 2005-07-15 | 2006-01-11 | 清华大学 | High-available distributed boundary gateway protocol system based on cluster router structure |
| CN103036717A (en) * | 2012-12-12 | 2013-04-10 | 北京邮电大学 | Consistency maintenance system and methods for distributed-type data |
| CN103544285A (en) * | 2013-10-28 | 2014-01-29 | 华为技术有限公司 | Data loading method and device |
| CN103631820A (en) * | 2012-08-27 | 2014-03-12 | 阿里巴巴集团控股有限公司 | Metadata management method and device of distributed file system |
| CN103729436A (en) * | 2013-12-27 | 2014-04-16 | 中国科学院信息工程研究所 | Distributed metadata management method and system |
| CN103942112A (en) * | 2013-01-22 | 2014-07-23 | 深圳市腾讯计算机系统有限公司 | Magnetic disk fault-tolerance method, device and system |
| CN104008152A (en) * | 2014-05-21 | 2014-08-27 | 华南理工大学 | Distributed file system architectural method supporting mass data access |
| CN104239418A (en) * | 2014-08-19 | 2014-12-24 | 天津南大通用数据技术股份有限公司 | Distributed lock method for supporting distributed database and distributed database system |
| CN107154915A (en) * | 2016-03-02 | 2017-09-12 | 阿里巴巴集团控股有限公司 | The method of defending distributed refusal service DDoS attack, apparatus and system |
-
2019
- 2019-03-13 CN CN201910189686.6A patent/CN111695018B/en active Active
Patent Citations (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JPH06318950A (en) * | 1993-05-10 | 1994-11-15 | Pfu Ltd | Control system for tree structure network |
| US20020055989A1 (en) * | 2000-11-08 | 2002-05-09 | Stringer-Calvert David W.J. | Methods and apparatus for scalable, distributed management of virtual private networks |
| US20050237926A1 (en) * | 2004-04-22 | 2005-10-27 | Fan-Tieng Cheng | Method for providing fault-tolerant application cluster service |
| CN1719831A (en) * | 2005-07-15 | 2006-01-11 | 清华大学 | High-available distributed boundary gateway protocol system based on cluster router structure |
| CN103631820A (en) * | 2012-08-27 | 2014-03-12 | 阿里巴巴集团控股有限公司 | Metadata management method and device of distributed file system |
| CN103036717A (en) * | 2012-12-12 | 2013-04-10 | 北京邮电大学 | Consistency maintenance system and methods for distributed-type data |
| CN103942112A (en) * | 2013-01-22 | 2014-07-23 | 深圳市腾讯计算机系统有限公司 | Magnetic disk fault-tolerance method, device and system |
| US20140229763A1 (en) * | 2013-01-22 | 2014-08-14 | Tencent Technology (Shenzhen) Company Limited | Disk fault tolerance method, device and system |
| CN103544285A (en) * | 2013-10-28 | 2014-01-29 | 华为技术有限公司 | Data loading method and device |
| CN103729436A (en) * | 2013-12-27 | 2014-04-16 | 中国科学院信息工程研究所 | Distributed metadata management method and system |
| CN104008152A (en) * | 2014-05-21 | 2014-08-27 | 华南理工大学 | Distributed file system architectural method supporting mass data access |
| CN104239418A (en) * | 2014-08-19 | 2014-12-24 | 天津南大通用数据技术股份有限公司 | Distributed lock method for supporting distributed database and distributed database system |
| CN107154915A (en) * | 2016-03-02 | 2017-09-12 | 阿里巴巴集团控股有限公司 | The method of defending distributed refusal service DDoS attack, apparatus and system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111695018B (en) | 2023-05-30 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN105376107B (en) | Terminal testing method and proxy server | |
| CN111163130B (en) | Network service system and data transmission method thereof | |
| CN103475682B (en) | File transfer method and file transfer equipment | |
| CN107181779B (en) | Method, device and system for processing access request | |
| CN111291079A (en) | Data query method and device | |
| CN109586969A (en) | Content distributing network disaster recovery method, device, computer equipment and storage medium | |
| CN105868231A (en) | Cache data updating method and device | |
| CN101018227A (en) | Data management device, storage medium having data management program, protocol switching device and method | |
| CN109246078B (en) | Data interaction method and server | |
| CN108880972B (en) | Information processing method, server and terminal | |
| CN104348848A (en) | Method, terminal equipment and server for managing pictures | |
| CN110704158A (en) | Method, apparatus and storage medium for forwarding access requests within a container cluster | |
| CN111159269B (en) | Data processing method, device and system | |
| CN114598735A (en) | Data processing method and system | |
| CN106571968B (en) | Service switching method and system | |
| CN110311855B (en) | User message processing method and device, electronic equipment and storage medium | |
| CN105721389A (en) | Multi-terminal equipment intercommunication data processing and service response method and device | |
| CN110708177B (en) | Exception handling method, system and device in distributed system | |
| CN111353136A (en) | Method and device for processing operation request | |
| CN110545453B (en) | Content distribution method, device and system of content distribution network | |
| CN111695018B (en) | Data processing method and device, distributed network system and computer equipment | |
| CN107203915B (en) | Data storage method and device | |
| CN104580672A (en) | Information processing method and electronic equipment | |
| CN111400327B (en) | Data synchronization method and device, electronic equipment and storage medium | |
| CN107071038A (en) | A kind of method and device of transmitting multimedia data |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20210907 Address after: Room 508, floor 5, building 4, No. 699, Wangshang Road, Changhe street, Binjiang District, Hangzhou City, Zhejiang Province Applicant after: Alibaba (China) Co.,Ltd. Address before: A four-storey 847 mailbox in Grand Cayman Capital Building, British Cayman Islands Applicant before: ALIBABA GROUP HOLDING Ltd. |
|
| TA01 | Transfer of patent application right | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20211118 Address after: 310000 No. 12, Zhuantang science and technology economic block, Xihu District, Hangzhou City, Zhejiang Province Applicant after: Aliyun Computing Co.,Ltd. Address before: 310052 room 508, 5th floor, building 4, No. 699 Wangshang Road, Changhe street, Binjiang District, Hangzhou City, Zhejiang Province Applicant before: Alibaba (China) Co.,Ltd. |
|
| TA01 | Transfer of patent application right | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |