CN112543222B - Data processing method and device, computer equipment and storage medium - Google Patents
Data processing method and device, computer equipment and storage medium Download PDFInfo
- Publication number
- CN112543222B CN112543222B CN202011257505.8A CN202011257505A CN112543222B CN 112543222 B CN112543222 B CN 112543222B CN 202011257505 A CN202011257505 A CN 202011257505A CN 112543222 B CN112543222 B CN 112543222B
- Authority
- CN
- China
- Prior art keywords
- data
- cluster
- data request
- target
- request
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/104—Peer-to-peer [P2P] networks
- H04L67/1044—Group management mechanisms
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/219—Managing data history or versioning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2282—Tablespace storage structures; Management thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1001—Protocols in which an application is distributed across nodes in the network for accessing one among a plurality of replicated servers
Abstract
The invention discloses a data request processing method, a data request processing device, computer equipment and a storage medium, wherein the method comprises the following steps: the method comprises the steps of receiving a data request sent by a client, carrying out request protocol analysis on the data request, determining a transmission protocol type corresponding to the data request according to an analysis result, carrying out request data analysis on the data request by using a preset data analysis module corresponding to the transmission protocol type, determining a target data table corresponding to the data request according to the analysis result, determining a target cluster where the target data table is located according to the relation between the preset data table and the cluster, and sending the data request to the target cluster so that the target cluster can execute corresponding operation on the target data table according to the data request. The invention realizes the compatibility of multiple protocols by configuring the analysis modules of different protocols at the gateway layer, can process different types of protocol requests and solves the problem of single type of forwarded protocol.
Description
Technical Field
The present invention relates to the field of multi-cluster management technologies, and in particular, to a data processing method and apparatus, a computer device, and a storage medium.
Background
Currently, there are two approaches to storage class multi-cluster management. One method is that a read-write request of a client is directly connected with a storage cluster, and a plurality of clusters are managed through a service management layer, such as cluster parameter setting, table creation, deletion and the like. Another method is to construct a gateway layer of multiple clusters, and the client requests are forwarded to the corresponding clusters through the gateway layer.
On one hand, the method for directly connecting the storage cluster by the client requests simplifies the complexity of multi-cluster management, and is convenient to use under the condition of small number of clusters. However, the method has the disadvantages that different users can be directly connected to the corresponding clusters, uniform traffic and security management and control are lacked, and the difference of different cluster versions to user requests cannot be shielded. With the development of services, when the number of clusters is gradually increased, the difficulty of cluster management by the method is gradually increased.
On the other hand, the gateway layer is used to receive the user's request and forward it to the corresponding cluster. Although the method solves the problem of uniform control of different versions of the cluster on the difference and the flow of the users, the method cannot realize multi-protocol compatibility, only can process the same type of protocol requests, and has a single forwarding protocol type. In addition, the method also lacks the safety control on the read-write request of the user, so that the large-data-volume write request also passes through the gateway layer, the read-write request is easily influenced, and even the gateway is overloaded and the like.
Therefore, a new data processing method is continuously proposed to solve the above problems.
Disclosure of Invention
In order to solve the problems in the prior art, embodiments of the present invention provide a data request processing method, an apparatus, a computer device, and a storage medium, so as to overcome the problems that in the prior art, a direct connection cluster of a user request cannot perform uniform management and control on a multi-cluster request, and forwarding the user request through a gateway layer cannot achieve multi-protocol compatibility, only can process requests of the same type of protocol, and the forwarded protocol type is single.
In order to solve one or more technical problems, the invention adopts the technical scheme that:
in a first aspect, a data request processing method is provided, which includes the following steps:
receiving a data request sent by a client, performing request protocol analysis on the data request, and determining a transmission protocol type corresponding to the data request according to an analysis result;
performing request data analysis on the data request by using a preset data analysis module corresponding to the transmission protocol type, and determining a target data table corresponding to the data request according to an analysis result;
determining a target cluster where the target data sheet is located according to a preset relation between the data sheet and the cluster;
and sending the data request to the target cluster so that the target cluster can execute corresponding operation on the target data table according to the data request.
Further, the determining, according to a preset relationship between the data table and the cluster, a target cluster where the target data table is located includes:
and inquiring whether the target data table exists in the cluster metadata, and if so, determining a target cluster where the target data table is located according to a preset relationship between the data table and the cluster.
Further, the data request includes user information, and the sending the data request to the target cluster includes:
extracting user information in the data request, inquiring a preset authority relation table according to the user information, determining whether the user has the operation authority of the target data table, and if so, sending the data request to the target cluster.
Further, the data request includes version information of the client, and the sending the data request to the target cluster includes:
extracting version information of the client in the data request;
determining whether the version of the client is matched with the version of the target cluster or not according to the version information of the client and a preset mapping relation table of the client version and the cluster version, and if so, directly sending the data request to the target cluster;
otherwise, adjusting parameters in the data request according to the version of the target cluster, and sending the data request with the adjusted parameters to the target cluster.
Further, the method further comprises:
and receiving real-time and/or offline data through a data integration module, and writing the real-time and/or offline data into a corresponding data table in the cluster according to a preset index relation.
Further, the method further comprises:
and monitoring the cluster metadata, and if the cluster metadata is monitored to be updated, updating the local cached cluster metadata.
Further, the respective operation includes at least one of a read operation and a write operation.
In a second aspect, there is provided a data request processing apparatus, the apparatus comprising:
the first analysis module is used for receiving a data request sent by a client, performing request protocol analysis on the data request, and determining a transmission protocol type corresponding to the data request according to an analysis result;
the second analysis module is used for performing request data analysis on the data request by utilizing a preset data analysis module corresponding to the transmission protocol type and determining a target data table corresponding to the data request according to an analysis result;
the cluster determining module is used for determining a target cluster where the target data sheet is located according to a preset relation between the data sheet and the cluster;
and the request forwarding module is used for sending the data request to the target cluster so that the target cluster can execute corresponding operation on the target data table according to the data request.
In a third aspect, a computer device is provided, which includes a memory, a processor, and a computer program stored on the memory and executable on the processor, and when the processor executes the computer program, the following steps are implemented:
receiving a data request sent by a client, performing request protocol analysis on the data request, and determining a transmission protocol type corresponding to the data request according to an analysis result;
performing request data analysis on the data request by using a preset data analysis module corresponding to the transmission protocol type, and determining a target data table corresponding to the data request according to an analysis result;
determining a target cluster where the target data sheet is located according to a preset relation between the data sheet and the cluster;
and sending the data request to the target cluster so that the target cluster can execute corresponding operation on the target data table according to the data request.
In a fourth aspect, there is provided a computer readable storage medium having a computer program stored thereon, which when executed by a processor, performs the steps of:
receiving a data request sent by a client, performing request protocol analysis on the data request, and determining a transmission protocol type corresponding to the data request according to an analysis result;
performing request data analysis on the data request by using a preset data analysis module corresponding to the transmission protocol type, and determining a target data table corresponding to the data request according to an analysis result;
determining a target cluster where the target data sheet is located according to a preset relation between the data sheet and the cluster;
and sending the data request to the target cluster so that the target cluster can execute corresponding operation on the target data table according to the data request.
The technical scheme provided by the embodiment of the invention has the following beneficial effects:
the data request processing method, the device, the computer equipment and the storage medium provided by the embodiment of the invention perform request protocol analysis on a data request by receiving the data request sent by a client, determine a transmission protocol type corresponding to the data request according to an analysis result, perform request data analysis on the data request by using a preset data analysis module corresponding to the transmission protocol type, determine a target data table corresponding to the data request according to the analysis result, determine a target cluster where the target data table is located according to the relation between the preset data table and the cluster, send the data request to the target cluster so that the target cluster can perform corresponding operation on the target data table according to the data request, realize multi-protocol compatibility by configuring analysis modules with different protocols on a gateway layer, and can process protocol requests with different types, the problem of single type of the forwarded protocol is solved;
further, according to the data request processing method, the data request processing device, the computer device, and the storage medium provided in the embodiments of the present invention, by extracting the user information in the data request, querying a preset authority relationship table according to the user information, determining whether the user has an operation authority of the target data table, and if yes, sending the data request to the target cluster, so as to implement management and control on the data request forwarded to the cluster, and reduce the pressure of the cluster;
further, the data request processing method, the data request processing device, the computer device, and the storage medium according to embodiments of the present invention receive real-time and/or offline data through the data integration module, write the real-time and/or offline data into a corresponding data table in a cluster according to a preset index relationship, and separate a large amount of data write requests from the data integration module to apply pressure to the gateway layer, thereby preventing the gateway layer from being overloaded and preventing read/write requests from affecting each other.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings needed to be used in the description of the embodiments will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.
FIG. 1 is an architecture diagram of a multi-cluster management system, shown in accordance with an exemplary embodiment;
FIG. 2 is a flow diagram illustrating a process for client data request through a gateway layer of a multi-cluster management system in accordance with an exemplary embodiment;
FIG. 3 is a flow diagram illustrating a data request processing method in accordance with an exemplary embodiment;
FIG. 4 is a schematic diagram illustrating a gateway layer service discovery flow through a multi-cluster management system in accordance with an exemplary embodiment;
FIG. 5 is a block diagram illustrating a data request processing device in accordance with an exemplary embodiment;
FIG. 6 is a schematic diagram illustrating an internal architecture of a computer device, according to an example embodiment.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
As described in the background art, in the storage type multi-cluster management method in the prior art, if a user request is directly connected to a cluster, there are problems that uniform traffic and security control are lacked and differences of different cluster versions to the user request cannot be shielded.
In order to solve the above problem, the embodiment of the present invention creatively provides a data request processing method, which separates a service discovery module, a gateway layer module, a cluster service management module, and the like from cluster management, and respectively enhances different modules. The gateway layer module supports service consistency discovery, multi-cluster gateways, multi-protocol version adaptation, flow control and the like, and the data integration module separates the pressure of a large number of data write requests on the gateway layer.
Example one
As shown in fig. 1, the present application provides a multi-cluster management system, which at least includes a cluster Service management module (Admin Service), a Service discovery module, a gateway layer module, a data integration module, a cluster, and the like. In the embodiment of the present invention, the cluster Service management module (Admin Service) is responsible for management work of multiple clusters, and is responsible for metadata operation of multiple clusters, and issues metadata in the Service discovery module, specifically, initiation, implementation, and state tracking of a cluster metadata change operation is provided, and after the metadata change operation is successfully implemented, metadata is updated to the Service discovery module, and is responsible for version maintenance, software management, resource management, configuration management, and the like of a cluster. The Service discovery module can be developed and realized by adopting an etcd Service discovery framework, and the etcd leader acquires the metadata state update of the multiple clusters of the Admin Service management module. Gateway layer modules include, but are not limited to, gateway. The gateway layer is used as a gateway layer of all the cluster services and is responsible for the work of request analysis, cluster service discovery, version adaptation, flow statistics, management and control and the like.
In the embodiment of the invention, load balancing is also arranged, and firstly, load balancing (LoadBalance) is used for flow access load balancing to be responsible for forwarding the request to a gateway layer (namely a gateway layer), so that the overall throughput capacity is improved. As a better implementation mode, the load balancing can adopt a DR mode, request responses are directly returned to the client, the server flow of the load balancing is obviously reduced, the running state of the back-end server is sensed by adopting a local minimum connection algorithm, and the dynamic expansion capacity of gateway is realized. The gateway layer is used as a gateway layer of all cluster services and is mainly responsible for the work of request analysis, cluster service discovery, version adaptation, flow statistics, management and control and the like.
Specifically, referring to fig. 2, a processing procedure of performing a client data request through the gateway layer of the multi-cluster management system is as follows:
the method comprises the following steps: receiving a data request sent by a client, performing request protocol analysis processing on the data request, and determining the transmission protocol type of the data request.
Specifically, the existing cluster management system that receives a user data request through a gateway layer (e.g., gateway layer) generally cannot achieve multi-protocol compatibility, and can only process the same type of protocol request, i.e., the type of protocol that can be forwarded is single. In order to solve the problem of single type of protocol forwarding, adaptation of at least two protocols including but not limited to http, tcp, etc. is implemented in the gateway layer (i.e. gateway layer). As a better example, in the embodiment of the present invention, the gateway layer is configured to implement adaptation of two protocols, http and tcp. After receiving a data request sent by a user through a client, firstly, performing request protocol analysis processing on the data request, and determining the transmission protocol type of the data request. In specific implementation, the protocol identifier carried in the data request can be analyzed, and the transmission protocol type corresponding to the data request is determined through the identifier. In addition, it should be noted that, in the embodiment of the present invention, the data request sent by the user through the client includes, but is not limited to, a data read request, a data write request, and the like.
Step two: and sending the data request to a corresponding data analysis module for data analysis processing.
Specifically, in the embodiment of the present invention, a data parsing module corresponding to the adapted protocol is also configured in the gateway layer (i.e., the gateway layer) in advance. Taking two protocols of http and tcp as examples, in the embodiment of the present invention, an http parsing module (such as an httpRequest parsing module) and a tcp parsing module (a transportAction parsing module) are also configured in the gateway layer (i.e., the gateway layer). If the data request analyzed in the above steps is an http request, forwarding the http request to an http request analysis module for data analysis, if the data request analyzed in the above steps is a tcp request, forwarding the tcp request to a transportAction analysis module for response analysis, and if the transmission protocol type corresponding to the data request analyzed in the above steps is unidentifiable by the gateway layer, directly feeding back an unidentifiable identification result to the client.
Step three: and analyzing and acquiring information such as a corresponding target data table from the data request by using a data analysis module.
Specifically, the data request is analyzed by using the corresponding data analysis module, and the target data table corresponding to the data request is obtained by analyzing the data request (i.e., the request body).
After the target data table corresponding to the data request is determined, whether the target data table exists in the metadata needs to be inquired. If the target data table exists in the metadata, further acquiring the cluster to which the target data table belongs, if the target data table does not exist in the metadata, directly returning information that the data table does not exist to the client, and ending the request at the moment. In specific implementation, the relationship between each data table and the cluster may be configured in advance, the relationship between the data table and the cluster at least includes the relationship between the data table and the cluster, and then the target cluster where the target data table is located is determined according to the relationship between the data table and the cluster.
Step four: and carrying out authority check on the target data table.
Specifically, after determining the target data table corresponding to the data request and the target cluster to which the target data table belongs, the authority check of the request of the target data table is required. The authority here refers to whether the user of the current data request has the operation authority of the target data table, including but not limited to read authority, write authority, etc. If the operation authority of the target data table exists, continuing to perform the next step, and if the operation authority of the target data table does not exist, directly returning information that the authority does not pass to the client side, and the like, wherein the request is ended at the moment.
Step five: and carrying out version adaptation verification on the data request.
Specifically, in order to ensure that the target cluster can process the data request, before sending the data request to the target cluster, version adaptation verification needs to be performed on the data request. In specific implementation, as a better example, a mapping relation table between a client version and a cluster version may be configured in advance, when a user sends a data request through a client, the data request may carry version information of the client, and when performing version adaptation verification, first, the version information of the client that may be carried in the data request is analyzed, and then, whether the version of the client is adapted to the version of a target cluster is determined according to the version information of the client and the mapping relation table between the client version and the cluster version. If the version of the client is not adapted to the version of the target cluster, the data request needs to be assembled into a request protocol compatible with the target cluster according to the version of the target cluster, that is, parameters in the data request need to be adjusted according to the version of the target cluster, so that the request parameters are adjusted to be consistent with the version of the target cluster.
Step six: and sending the data request to the target cluster.
Specifically, if the version of the client is adapted to the version of the target cluster, the data request may be directly sent to the target cluster, otherwise, parameters in the data request need to be adjusted according to the version of the target cluster, so that the request parameters are adjusted to be consistent with the version of the target cluster, and then the data request with the parameters adjusted is sent to the target cluster.
Example two
Fig. 3 is a flow chart illustrating a data request processing method according to an exemplary embodiment, and referring to fig. 3, the method includes the steps of:
s1: receiving a data request sent by a client, performing request protocol analysis on the data request, and determining a transmission protocol type corresponding to the data request according to an analysis result.
Specifically, the multi-cluster management system in the embodiment of the present invention receives, through the gateway layer, a data request sent by a user through the client, and forwards the data request to the corresponding cluster. The gateway layer supports processing of at least two transport protocols including, but not limited to, http, tcp, and the like. When a data request sent by a client is received, first, request protocol analysis is performed on the data request, an analysis result is obtained, the analysis result includes but is not limited to information such as a protocol identifier (such as a request header), and a transmission protocol type corresponding to the data request is determined through the protocol identifier.
S2: and performing request data analysis on the data request by using a preset data analysis module corresponding to the transmission protocol type, and determining a target data table corresponding to the data request according to an analysis result.
Specifically, in the embodiment of the present invention, the multi-cluster management system receives, through the gateway layer, a data request sent by a user through the client, and forwards the data request to the corresponding cluster. The gateway layer supports processing of at least two transport protocols including, but not limited to, http, tcp, and the like. When a data request sent by a client is received, first, request protocol analysis is performed on the data request, an analysis result is obtained, the analysis result includes but is not limited to information such as a protocol identifier (such as a request header), and a transmission protocol type corresponding to the data request is determined through the protocol identifier.
S3: and determining a target cluster where the target data sheet is located according to a preset relation between the data sheet and the cluster.
Specifically, after the target data table corresponding to the data request is determined, the target cluster where the target data table is located needs to be determined. In specific implementation, a relationship between a data table and a cluster may be preconfigured, the cluster in which each data table is located is recorded in the relationship, and the target cluster in which the target data table is located is determined by querying the relationship between the data table and the cluster.
S4: and sending the data request to the target cluster so that the target cluster can execute corresponding operation on the target data table according to the data request.
Specifically, in the embodiment of the present invention, the corresponding operation includes, but is not limited to, a read operation, a write operation, and the like.
As a preferred implementation manner, in the embodiment of the present invention, the determining, according to a preset relationship between the data table and the cluster, a target cluster where the target data table is located includes:
and inquiring whether the target data table exists in the cluster metadata, and if so, determining a target cluster where the target data table is located according to a preset relationship between the data table and the cluster.
Specifically, as an example, when determining a target cluster where the target data table is located, it may be queried whether the target data table exists in the cluster metadata, if the target data table exists in the cluster metadata, further obtain a cluster to which the target data table belongs, and if the target data table does not exist in the cluster metadata, directly return, and the request is ended.
As a preferred implementation manner, in an embodiment of the present invention, the sending the data request to the target cluster includes:
extracting user information in the data request, inquiring a preset authority relation table according to the user information, determining whether the user has the operation authority of the target data table, and if so, sending the data request to the target cluster.
As a preferred implementation manner, in an embodiment of the present invention, the data request includes version information of the client, and the sending the data request to the target cluster includes:
extracting version information of the client in the data request;
determining whether the version of the client is matched with the version of the target cluster or not according to the version information of the client and a preset mapping relation table of the client version and the cluster version, and if so, directly sending the data request to the target cluster;
otherwise, adjusting parameters in the data request according to the version of the target cluster, and sending the data request with the adjusted parameters to the target cluster.
As a preferred implementation manner, in an embodiment of the present invention, the method further includes:
and receiving real-time and/or offline data through a data integration module, and writing the real-time and/or offline data into a corresponding data table in the cluster according to a preset index relation.
Specifically, in order to reduce the pressure of a large amount of data write requests on the gateway layer, the data integration module is arranged in the multi-cluster management system provided by the invention, and the data integration module is used for writing corresponding data into the corresponding cluster, so that the pressure of the large amount of data write requests on the gateway is separated. Further referring to fig. 1, in a specific implementation, the data integration module pre-configures a data writing task, and requests an index relationship of a cluster from a cluster Service management module (Admin Service) when the data writing task is configured. And executing the data writing task regularly, receiving real-time and/or offline data sent by data tools such as kafka and Hive, and writing the real-time and/or offline data into a corresponding data table in the cluster according to a preset index relation.
As a preferred implementation manner, in an embodiment of the present invention, the method further includes:
and monitoring the cluster metadata, and if the cluster metadata is monitored to be updated, updating the local cached cluster metadata.
Specifically, as further shown in fig. 1, the multi-cluster management system provided by the present invention is further provided with a Service discovery module, and during specific implementation, the Service discovery module can be developed and implemented by using an etcd Service discovery framework, and an etcd leader obtains the metadata state update of the multi-cluster of the Admin Service management module.
Fig. 4 is a schematic diagram illustrating a gateway layer service discovery process through a multi-cluster management system according to an exemplary embodiment, and referring to fig. 4, a service discovery module uses the service discovery service of etcd, and monitors the state of cluster management metadata to implement change distribution of cluster and table data. In specific implementation, the cluster service management module is responsible for acquiring node expansion and contraction capacity of each cluster and metadata change of a table level, updating the changed metadata to the service discovery module, and the gateway layer is connected with the service discovery module and monitors whether the cluster metadata change or not, and if the cluster metadata change, the latest metadata are cached in service discovery of the gateway layer. As a better example, when the gateway layer updates the metadata, a distributed lock is used in the caching process to ensure the consistency of the metadata of each gateway layer instance.
As a preferred implementation manner, in the embodiment of the present invention, the corresponding operation includes at least one of a read operation and a write operation.
Fig. 5 is a schematic structural diagram illustrating a data request processing apparatus according to an exemplary embodiment, the apparatus including:
the first analysis module is used for receiving a data request sent by a client, performing request protocol analysis on the data request, and determining a transmission protocol type corresponding to the data request according to an analysis result;
the second analysis module is used for performing request data analysis on the data request by utilizing a preset data analysis module corresponding to the transmission protocol type and determining a target data table corresponding to the data request according to an analysis result;
the cluster determining module is used for determining a target cluster where the target data sheet is located according to a preset relation between the data sheet and the cluster;
and the request forwarding module is used for sending the data request to the target cluster so that the target cluster can execute corresponding operation on the target data table according to the data request.
As a preferred implementation manner, in the embodiment of the present invention, the cluster determining module is specifically configured to:
and inquiring whether the target data table exists in the cluster metadata, and if so, determining a target cluster where the target data table is located according to a preset relationship between the data table and the cluster.
As a preferred implementation manner, in the embodiment of the present invention, the request forwarding module is specifically configured to:
extracting user information in the data request, inquiring a preset authority relation table according to the user information, determining whether the user has the operation authority of the target data table, and if so, sending the data request to the target cluster.
As a preferred implementation manner, in the embodiment of the present invention, the request forwarding module is further configured to:
extracting version information of the client in the data request;
determining whether the version of the client is matched with the version of the target cluster or not according to the version information of the client and a preset mapping relation table of the client version and the cluster version, and if so, directly sending the data request to the target cluster;
otherwise, adjusting parameters in the data request according to the version of the target cluster, and sending the data request with the adjusted parameters to the target cluster.
As a preferred implementation manner, in an embodiment of the present invention, the apparatus further includes:
and the data writing module is used for receiving the real-time and/or offline data through the data integration module and writing the real-time and/or offline data into the corresponding data table in the cluster according to a preset index relationship.
As a preferred implementation manner, in an embodiment of the present invention, the apparatus further includes:
and the service discovery module is used for monitoring the cluster metadata, and updating the local cached cluster metadata if the update of the cluster metadata is monitored.
As a preferred implementation manner, in the embodiment of the present invention, the corresponding operation includes at least one of a read operation and a write operation.
Fig. 6 is a schematic diagram illustrating an internal configuration of a computer device according to an exemplary embodiment, which includes a processor, a memory, and a network interface connected through a system bus, as shown in fig. 6. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device comprises a nonvolatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of an operating system and computer programs in the non-volatile storage medium. The network interface of the computer device is used for communicating with an external terminal through a network connection. The computer program is executed by a processor to implement a method of optimization of an execution plan.
Those skilled in the art will appreciate that the configuration shown in fig. 6 is a block diagram of only a portion of the configuration associated with aspects of the present invention and is not intended to limit the computing devices to which aspects of the present invention may be applied, and that a particular computing device may include more or less components than those shown, or may combine certain components, or have a different arrangement of components.
As a preferred implementation manner, in an embodiment of the present invention, the computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and when the processor executes the computer program, the following steps are implemented:
receiving a data request sent by a client, performing request protocol analysis on the data request, and determining a transmission protocol type corresponding to the data request according to an analysis result;
performing request data analysis on the data request by using a preset data analysis module corresponding to the transmission protocol type, and determining a target data table corresponding to the data request according to an analysis result;
determining a target cluster where the target data sheet is located according to a preset relation between the data sheet and the cluster;
and sending the data request to the target cluster so that the target cluster can execute corresponding operation on the target data table according to the data request.
As a preferred implementation manner, in the embodiment of the present invention, when the processor executes the computer program, the following steps are further implemented:
and inquiring whether the target data table exists in the cluster metadata, and if so, determining a target cluster where the target data table is located according to a preset relationship between the data table and the cluster.
As a preferred implementation manner, in the embodiment of the present invention, when the processor executes the computer program, the following steps are further implemented:
extracting user information in the data request, inquiring a preset authority relation table according to the user information, determining whether the user has the operation authority of the target data table, and if so, sending the data request to the target cluster.
As a preferred implementation manner, in the embodiment of the present invention, when the processor executes the computer program, the following steps are further implemented:
extracting version information of the client in the data request;
determining whether the version of the client is matched with the version of the target cluster or not according to the version information of the client and a preset mapping relation table of the client version and the cluster version, and if so, directly sending the data request to the target cluster;
otherwise, adjusting parameters in the data request according to the version of the target cluster, and sending the data request with the adjusted parameters to the target cluster.
As a preferred implementation manner, in the embodiment of the present invention, when the processor executes the computer program, the following steps are further implemented:
and receiving real-time and/or offline data through a data integration module, and writing the real-time and/or offline data into a corresponding data table in the cluster according to a preset index relation.
As a preferred implementation manner, in the embodiment of the present invention, when the processor executes the computer program, the following steps are further implemented:
and monitoring the cluster metadata, and if the cluster metadata is monitored to be updated, updating the local cached cluster metadata.
In an embodiment of the present invention, a computer-readable storage medium is further provided, on which a computer program is stored, and when the computer program is executed by a processor, the computer program implements the following steps:
receiving a data request sent by a client, performing request protocol analysis on the data request, and determining a transmission protocol type corresponding to the data request according to an analysis result;
performing request data analysis on the data request by using a preset data analysis module corresponding to the transmission protocol type, and determining a target data table corresponding to the data request according to an analysis result;
determining a target cluster where the target data sheet is located according to a preset relation between the data sheet and the cluster;
and sending the data request to the target cluster so that the target cluster can execute corresponding operation on the target data table according to the data request.
As a preferred implementation manner, in the embodiment of the present invention, when executed by the processor, the computer program further implements the following steps:
and inquiring whether the target data table exists in the cluster metadata, and if so, determining a target cluster where the target data table is located according to a preset relationship between the data table and the cluster.
As a preferred implementation manner, in the embodiment of the present invention, when executed by the processor, the computer program further implements the following steps:
extracting user information in the data request, inquiring a preset authority relation table according to the user information, determining whether the user has the operation authority of the target data table, and if so, sending the data request to the target cluster.
As a preferred implementation manner, in the embodiment of the present invention, when executed by the processor, the computer program further implements the following steps:
extracting version information of the client in the data request;
determining whether the version of the client is matched with the version of the target cluster or not according to the version information of the client and a preset mapping relation table of the client version and the cluster version, and if so, directly sending the data request to the target cluster;
otherwise, adjusting parameters in the data request according to the version of the target cluster, and sending the data request with the adjusted parameters to the target cluster.
As a preferred implementation manner, in the embodiment of the present invention, when executed by the processor, the computer program further implements the following steps:
and receiving real-time and/or offline data through a data integration module, and writing the real-time and/or offline data into a corresponding data table in the cluster according to a preset index relation.
As a preferred implementation manner, in the embodiment of the present invention, when executed by the processor, the computer program further implements the following steps:
and monitoring the cluster metadata, and if the cluster metadata is monitored to be updated, updating the local cached cluster metadata.
In summary, the technical solution provided by the embodiment of the present invention has the following beneficial effects:
the data request processing method, the device, the computer equipment and the storage medium provided by the embodiment of the invention perform request protocol analysis on a data request by receiving the data request sent by a client, determine a transmission protocol type corresponding to the data request according to an analysis result, perform request data analysis on the data request by using a preset data analysis module corresponding to the transmission protocol type, determine a target data table corresponding to the data request according to the analysis result, determine a target cluster where the target data table is located according to the relation between the preset data table and the cluster, send the data request to the target cluster so that the target cluster can perform corresponding operation on the target data table according to the data request, realize multi-protocol compatibility by configuring analysis modules with different protocols on a gateway layer, and can process protocol requests with different types, the problem of single type of the forwarded protocol is solved;
further, according to the data request processing method, the data request processing device, the computer device and the storage medium provided by the embodiment of the invention, by extracting the user information in the data request, querying a preset authority relationship table according to the user information, determining whether the user has the operation authority of the target data table, and if so, sending the data request to the target cluster, so that the control on the data request forwarded to the cluster is realized, and the pressure of the cluster is reduced;
further, the data request processing method, the data request processing device, the computer device, and the storage medium according to embodiments of the present invention receive real-time and/or offline data through the data integration module, write the real-time and/or offline data into a corresponding data table in a cluster according to a preset index relationship, and separate a large amount of data write requests from the data integration module to apply pressure to the gateway layer, thereby preventing the gateway layer from being overloaded and preventing read/write requests from affecting each other.
It should be noted that: the data request processing apparatus provided in the foregoing embodiment is only illustrated by dividing the functional modules when triggering the data processing service, and in practical applications, the function allocation may be completed by different functional modules according to needs, that is, the internal structure of the apparatus is divided into different functional modules to complete all or part of the functions described above. In addition, the data request processing apparatus and the data request processing method provided in the above embodiments belong to the same concept, that is, the apparatus is based on the data request processing method, and the specific implementation process thereof is described in the method embodiments, and is not described herein again.
It will be understood by those skilled in the art that all or part of the steps for implementing the above embodiments may be implemented by hardware, or may be implemented by a program instructing relevant hardware, where the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a magnetic disk or an optical disk, etc.
The above description is only for the purpose of illustrating the preferred embodiments of the present invention and is not to be construed as limiting the invention, and any modifications, equivalents, improvements and the like that fall within the spirit and principle of the present invention are intended to be included therein.
Claims (10)
1. A method for processing data requests, the method comprising the steps of:
receiving a data request sent by a client, performing request protocol analysis on the data request, and determining a transmission protocol type corresponding to the data request according to an analysis result;
performing request data analysis on the data request by using a preset data analysis module corresponding to the transmission protocol type, and determining a target data table corresponding to the data request according to an analysis result;
determining a target cluster where the target data sheet is located according to a preset relation between the data sheet and the cluster;
and sending the data request to the target cluster so that the target cluster can execute corresponding operation on the target data table according to the data request.
2. The data request processing method of claim 1, wherein the determining the target cluster where the target data table is located according to a preset data table and cluster relationship comprises:
and inquiring whether the target data table exists in the cluster metadata, and if so, determining a target cluster where the target data table is located according to a preset relationship between the data table and the cluster.
3. The data request processing method according to claim 1 or 2, wherein the data request includes user information, and the sending the data request to the target cluster includes:
extracting user information in the data request, inquiring a preset authority relation table according to the user information, determining whether the user has the operation authority of the target data table, and if so, sending the data request to the target cluster.
4. The data request processing method according to claim 1 or 2, wherein the data request includes version information of the client, and the sending the data request to the target cluster includes:
extracting version information of the client in the data request;
determining whether the version of the client is matched with the version of the target cluster or not according to the version information of the client and a preset mapping relation table of the client version and the cluster version, and if so, directly sending the data request to the target cluster;
otherwise, adjusting parameters in the data request according to the version of the target cluster, and sending the data request with the adjusted parameters to the target cluster.
5. The data request processing method according to claim 1 or 2, wherein the method further comprises:
and receiving real-time and/or offline data through a data integration module, and writing the real-time and/or offline data into a corresponding data table in the cluster according to a preset index relation.
6. The data request processing method of claim 2, wherein the method further comprises:
and monitoring the cluster metadata, and if the cluster metadata is monitored to be updated, updating the local cached cluster metadata.
7. The data request processing method of claim 1 or 2, wherein the respective operation comprises at least one of a read operation and a write operation.
8. A data request processing apparatus, characterized in that the apparatus comprises:
the first analysis module is used for receiving a data request sent by a client, performing request protocol analysis on the data request, and determining a transmission protocol type corresponding to the data request according to an analysis result;
the second analysis module is used for performing request data analysis on the data request by utilizing a preset data analysis module corresponding to the transmission protocol type and determining a target data table corresponding to the data request according to an analysis result;
the cluster determining module is used for determining a target cluster where the target data sheet is located according to a preset relation between the data sheet and the cluster;
and the request forwarding module is used for sending the data request to the target cluster so that the target cluster can execute corresponding operation on the target data table according to the data request.
9. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the steps of the method of any of claims 1 to 7 are implemented when the computer program is executed by the processor.
10. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the method of any one of claims 1 to 7.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011257505.8A CN112543222B (en) | 2020-11-11 | 2020-11-11 | Data processing method and device, computer equipment and storage medium |
| CA3138764A CA3138764A1 (en) | 2020-11-11 | 2021-11-12 | Data processing method, device, computer equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011257505.8A CN112543222B (en) | 2020-11-11 | 2020-11-11 | Data processing method and device, computer equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112543222A CN112543222A (en) | 2021-03-23 |
| CN112543222B true CN112543222B (en) | 2022-07-05 |
Family
ID=75015064
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011257505.8A Active CN112543222B (en) | 2020-11-11 | 2020-11-11 | Data processing method and device, computer equipment and storage medium |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN112543222B (en) |
| CA (1) | CA3138764A1 (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113641503A (en) * | 2021-09-01 | 2021-11-12 | 上海联蔚盘云科技有限公司 | Multi-cloud multi-cluster Kubernetes management system, method and equipment |
| CN115065725B (en) * | 2022-05-12 | 2024-01-12 | 北京奇艺世纪科技有限公司 | Data processing method, device, electronic equipment and storage medium |
| CN115250297A (en) * | 2022-06-28 | 2022-10-28 | 合肥移顺信息技术有限公司 | Data analysis method, device, terminal, medium and program product for digital subscriber line network |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108289060A (en) * | 2017-01-09 | 2018-07-17 | 腾讯科技（深圳）有限公司 | A kind of data processing method and device based on RPC service |
| US20200133949A1 (en) * | 2018-10-26 | 2020-04-30 | Alibaba Group Holding Limited | Data processing method, apparatus, and device |
| CN111124299A (en) * | 2019-12-17 | 2020-05-08 | 网易传媒科技（北京）有限公司 | Data storage management method, device, equipment, system and storage medium |
-
2020
- 2020-11-11 CN CN202011257505.8A patent/CN112543222B/en active Active
-
2021
- 2021-11-12 CA CA3138764A patent/CA3138764A1/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108289060A (en) * | 2017-01-09 | 2018-07-17 | 腾讯科技（深圳）有限公司 | A kind of data processing method and device based on RPC service |
| US20200133949A1 (en) * | 2018-10-26 | 2020-04-30 | Alibaba Group Holding Limited | Data processing method, apparatus, and device |
| CN111124299A (en) * | 2019-12-17 | 2020-05-08 | 网易传媒科技（北京）有限公司 | Data storage management method, device, equipment, system and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112543222A (en) | 2021-03-23 |
| CA3138764A1 (en) | 2022-05-11 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112543222B (en) | Data processing method and device, computer equipment and storage medium | |
| CN108965381B (en) | Nginx-based load balancing implementation method and device, computer equipment and medium | |
| CN110290189B (en) | Container cluster management method, device and system | |
| KR100629057B1 (en) | Remote dynamic configuration of a web server to facilitate capacity on demand | |
| US8504733B1 (en) | Subtree for an aggregation system | |
| US8069224B2 (en) | Method, equipment and system for resource acquisition | |
| CN112532675B (en) | Method, device and medium for establishing network edge computing system | |
| WO2022002209A1 (en) | Data transmission method, proxy server, storage medium, and electronic device | |
| CN105357296A (en) | Elastic caching system based on Docker cloud platform | |
| CN105049268A (en) | Distributed computing resource allocation system and task processing method | |
| US7937704B2 (en) | Distributed computer | |
| WO2004036344A2 (en) | System and method for the optimization of database | |
| KR100606025B1 (en) | Network management apparatus and method based on simple network management protocol | |
| CN110297867B (en) | Database cluster operation method and system based on domestic CPU and distributed container cluster | |
| CN113835836B (en) | System, method, computer device and medium for dynamic publishing container service | |
| CN113326155A (en) | Information processing method, device, system and storage medium | |
| US20220318071A1 (en) | Load balancing method and related device | |
| US7966394B1 (en) | Information model registry and brokering in virtualized environments | |
| CN112506755A (en) | Log collection method and device, computer equipment and storage medium | |
| CN112445774A (en) | Distributed shared file system and data processing method thereof | |
| CN106547790B (en) | Relational database service system | |
| CN112052227A (en) | Data change log processing method and device and electronic equipment | |
| CN116805946A (en) | Message request processing method and device, electronic equipment and storage medium | |
| CN113177179B (en) | Data request connection management method, device, equipment and storage medium | |
| US9015371B1 (en) | Method to discover multiple paths to disk devices cluster wide |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |