CN112988874A - Data processing method, system, computing device and readable storage medium - Google Patents
Data processing method, system, computing device and readable storage medium Download PDFInfo
- Publication number
- CN112988874A CN112988874A CN202110377822.1A CN202110377822A CN112988874A CN 112988874 A CN112988874 A CN 112988874A CN 202110377822 A CN202110377822 A CN 202110377822A CN 112988874 A CN112988874 A CN 112988874A
- Authority
- CN
- China
- Prior art keywords
- database
- target
- data
- data request
- proxy router
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 25
- 238000000034 method Methods 0.000 claims abstract description 35
- 238000012545 processing Methods 0.000 claims abstract description 26
- 230000004044 response Effects 0.000 claims abstract description 7
- 238000004891 communication Methods 0.000 claims description 19
- 238000013507 mapping Methods 0.000 claims description 4
- 230000008569 process Effects 0.000 description 8
- 238000010586 diagram Methods 0.000 description 6
- 238000004364 calculation method Methods 0.000 description 4
- 238000011161 development Methods 0.000 description 4
- 230000006870 function Effects 0.000 description 4
- 238000012360 testing method Methods 0.000 description 4
- 230000002093 peripheral effect Effects 0.000 description 3
- 230000008901 benefit Effects 0.000 description 2
- 238000013500 data storage Methods 0.000 description 2
- 230000007723 transport mechanism Effects 0.000 description 2
- 238000012384 transportation and delivery Methods 0.000 description 2
- 238000013523 data management Methods 0.000 description 1
- 238000013506 data mapping Methods 0.000 description 1
- 230000001419 dependent effect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000007667 floating Methods 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 238000007726 management method Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2453—Query optimisation
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L45/00—Routing or path finding of packets in data switching networks
- H04L45/54—Organization of routing tables
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/56—Provisioning of proxy services
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/60—Scheduling or organising the servicing of application requests, e.g. requests for application data transmissions using the analysis and optimisation of the required network resources
- H04L67/63—Routing a service request depending on the request content or context
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computational Linguistics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a data processing method, which is suitable for being executed in a data processing system, wherein the data processing system comprises a client, an agent router, a database manager and a database, and the method comprises the following steps: the client sends a data request to the proxy router, wherein the data request comprises target data and a target database corresponding to the target data; the proxy router acquires a target database in the data request and acquires a database manager corresponding to the target database according to the target database; the proxy router sends the data request to a database manager corresponding to the target database; and the database manager completes data operation in the corresponding database according to the data request and returns a response message. The invention also discloses a corresponding system, a computing device and a readable storage medium.
Description
Technical Field
The present invention relates to the field of data storage, and in particular, to a data processing method, system, computing device, and readable storage medium.
Background
With the rapid development of informatization and IT technology, enterprise data is exponentially exploded, the traditional centralized storage and calculation mode cannot meet the increasing business requirements of enterprises, Hadoop arises, the distributed architecture design effectively solves the problem of distributed storage and calculation of data, developers can write MapReduce programs to conveniently realize mass data processing through a Hadoop platform, and a Hadoop cluster can run on cheap commercial computer hardware, so that the cost investment of the enterprises is greatly saved.
Hive is used as an important member in Hadoop ecology, data storage of the Hive depends on Hadoop, and meanwhile, SQL-like language is provided for processing data, so that a user does not need to write a complicated MapReduce program to complete data calculation, and the use threshold of the user is reduced. When a Hive query is operated, firstly, metadata information related to the query is acquired through a MetaStore service, the metadata information required by the MetaStore service is actually stored in a database (which can be MySQL, Derby and other databases), distributed data calculation can be executed in a Hadoop cluster according to the acquired data mapping rules after the metadata is queried, and finally, a query result is returned. When the Hadoop cluster is very large in scale and a large number of Hive job queries are executed concurrently, if only a single MetaStore instance is deployed, the access concurrency of the MetaStore becomes very high, the metadata query service becomes a performance bottleneck of the whole system, the operation efficiency of query jobs is greatly influenced, and even the MetaStore crash occurs in severe cases, so that the whole service is unavailable. However, the native Hive component supports the horizontal extension of the MetaStore instances, and a plurality of sets of MetaStore instances can be configured to realize load balancing. However, the metadata information of all Hive tables can only be stored in the same Mysql database, all MetaStore instances need to read the metadata from the same Mysql database, and the Mysql service cannot be expanded horizontally, and does not have the distributed storage capacity of the metadata information. When the amount of metadata information data becomes very large, there is also a performance bottleneck in the MetaStore's access to the Mysql database. Too many metadata records in Mysql lead to too long time consumption of single query and too high query load of Mysql in business peak period, and become another big performance killer of the current Hive architecture.
Disclosure of Invention
To this end, the present invention provides a data processing method, system, computing device and readable storage medium in an attempt to solve, or at least alleviate, the problems identified above.
According to one aspect of the present invention, there is provided a data processing method adapted to be executed in a data processing system, the data processing system comprising a client, a proxy router, a database manager, a database, the method comprising: the client sends a data request to the proxy router, wherein the data request comprises target data and a target database corresponding to the target data; the proxy router acquires a target database in the data request and acquires a database manager corresponding to the target database according to the target database; the proxy router sends the data request to a database manager corresponding to the target database; and the database manager completes data operation in the corresponding database according to the data request and returns a response message.
Optionally, in the data processing method according to the present invention, the client stores a proxy router list, the proxy router list stores address information of the proxy router and the proxy router, and the sending of the data request to the proxy router by the client includes: the client selects a target proxy router from the proxy router list according to a first preset rule, and sends the data request to the target proxy router according to the address information of the target proxy router.
Optionally, in the data processing method according to the present invention, the data processing system further includes a routing information database, a mapping relationship between the database and a database manager is stored in the routing information database, all the agent routers are in communication connection with the routing information database, and obtaining the database manager corresponding to the target database according to the target database includes: and the agent router queries the routing information database according to the target database and acquires a database manager corresponding to the target database.
Optionally, in the data processing method according to the present invention, the database manager includes a plurality of metastores, and the sending, by the proxy router, the data request to the database manager corresponding to the target database includes: selecting a target MetaStore from a plurality of MetaStores according to a second preset rule; and sending the data request to the target MetaStore.
Optionally, in the data processing method according to the present invention, the routing table further includes a newly created database identification field, and when the data request is to create a new database, acquiring, according to the target database, a database manager corresponding to the target database includes: a database manager is obtained that identifies the newly created database as genuine.
Optionally, in the data processing method according to the present invention, the proxy router supports kerberos, and the method further includes: the client completes authentication with the agent router based on kerberos; the proxy router completes authentication with the database manager based on kerberos.
Optionally, in the data processing method according to the present invention, the database is configured as a master-slave, and includes a master database and a slave database, and the master database and the slave database are consistent in data.
According to another aspect of the present invention, a data processing system is provided, which includes a client, a proxy router, a database manager, and a database, wherein the client is adapted to send a data request to the proxy router, the data request includes target data and a target database corresponding to the target data; the agent router is suitable for acquiring a target database in the data request and acquiring a database manager corresponding to the target database according to the target database; the data request is sent to a database manager corresponding to the target database; the database manager is suitable for finishing data operation in a corresponding database according to the data request and returning a response message; a database adapted to store target data.
According to yet another aspect of the invention, there is provided a computing device comprising: at least one processor; and a memory storing program instructions, wherein the program instructions are configured to be executed by the at least one processor, the program instructions comprising instructions for performing the data processing method as above.
According to still another aspect of the present invention, there is provided a readable storage medium storing program instructions which, when read and executed by a computing device, cause the computing device to execute the data processing method as above.
According to the data processing method of the invention, by adding the proxy router in the data processing system, the client firstly sends the data request to the proxy router, the proxy router sends the data request to the corresponding database manager according to the data request information, the database manager completes the corresponding operations of data query, database creation and the like according to the data request, and returns response information, different data managers are connected with different databases, a plurality of databases are connected through a plurality of database managers, therefore, the transverse expansion of the database layer is realized, the system performance bottleneck caused by the rapid increase of the service data is eliminated, the data query time can be shortened, the execution efficiency of the service operation is improved, the system performance bottleneck caused by the increase of the source data is eliminated, the system stability is improved, and the potential safety hazard in the system operation process is eliminated.
Furthermore, the agent router shares the same routing information database, so that unified management of all agent routing service configurations is realized, maintenance of routing information is facilitated, and meanwhile, the agent router supports kerberos, so that an access user can be authenticated, and tampering of stored data by a malicious user is prevented.
Drawings
To the accomplishment of the foregoing and related ends, certain illustrative aspects are described herein in connection with the following description and the annexed drawings, which are indicative of various ways in which the principles disclosed herein may be practiced, and all aspects and equivalents thereof are intended to be within the scope of the claimed subject matter. The above and other objects, features and advantages of the present disclosure will become more apparent from the following detailed description read in conjunction with the accompanying drawings. Throughout this disclosure, like reference numerals generally refer to like parts or elements.
FIG. 1 shows a schematic diagram of a data processing system 100, in accordance with one embodiment of the present invention;
FIG. 2 illustrates a block diagram of a computing device 200, according to one embodiment of the invention;
FIG. 3 shows a flow diagram of a data processing method 300 according to one embodiment of the invention.
Detailed Description
Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be embodied in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.
FIG. 1 shows a schematic diagram of a data processing system 100, according to one embodiment of the invention. As shown in FIG. 1, the data processing system 100 includes clients 110 (including client 110-1, client 110-2 … … client 110-n), proxy routers 120 (including proxy router 120-1, proxy router 120-2 … … proxy router 120-n), routing information database 130, database manager 140 (including database manager 140-1, database manager 140-2 … … database manager 140-n), and database 150 (including database 150-1, database 150-2 … … database 150-n). It should be noted that the content recommendation system 100 in fig. 1 is only exemplary, and in a specific practical situation, the numbers of the proxy router 120, the database manager 140, and the database 150 may be set according to a specific requirement, and the present invention is not limited thereto.
The client 110 is generally an application server for running an application program, the application server may be a debugging machine used by a development tester in a development stage, or may be an execution server after the application program is online, the client 110 may complete operations such as corresponding data query, search, database creation and the like by sending a data request, and the used query service may be Hiveserver2 or Presto, which is not limited in the present invention.
The proxy router 120 is used to complete forwarding of data requests, a data processing system includes a plurality of proxy routers, which can effectively balance data request information of clients, and all the proxy routers are communicatively connected to the routing information database 130 to forward data requests to the corresponding database manager 140 according to the routing information in the routing information database 130.
The routing information database 130 is used for storing routing information, the routing information includes a mapping relationship between the database and a database manager, all the agent routers acquire the routing information through the same routing information database, so that the routing information can be managed in a unified manner, and the condition that one piece of routing information is maintained on each agent router is avoided. The routing information database 130 may be implemented by any type of database, such as MySQL, ACCESS, Oracle, etc., which is not limited in the present invention.
The database manager 140 is connected to a corresponding database, according to an embodiment of the present invention, in the Hive database, one database manager includes a plurality of MetaStore instances, which can achieve load balancing and improve the data query process, and in the data query process, one MetaStore is selected according to a preset rule to complete the query operation.
The database 150 is used for storing business data, and may be implemented as a MySQL database or a Derby database, each database corresponds to a fixed database manager, the database 150 generally performs master-slave configuration, and keeps data synchronization between the master database and the slave database, and the problem of data query failure when the master database is down can be effectively avoided by performing the master-slave configuration.
All of the devices in the data processing system shown in FIG. 1 may be implemented by a computing device, and FIG. 2 illustrates a block diagram of a computing device 200, according to one embodiment of the invention. It should be noted that the computing device 200 shown in fig. 2 is only an example, and in practice, the computing device for implementing the data processing method of the present invention may be any type of device, and the hardware configuration thereof may be the same as that of the computing device 200 shown in fig. 2 or different from that of the computing device 200 shown in fig. 2. In practice, the computing device for implementing the data processing method of the present invention may add or delete hardware components of the computing device 200 shown in fig. 2, and the present invention does not limit the specific hardware configuration of the computing device.
As shown in FIG. 2, in a basic configuration 202, a computing device 200 typically includes a system memory 206 and one or more processors 204. A memory bus 208 may be used for communication between the processor 204 and the system memory 206.
Depending on the desired configuration, the processor 204 may be any type of processing, including but not limited to: a microprocessor (μ P), a microcontroller (μ C), a Digital Signal Processor (DSP), or any combination thereof. The processor 204 may include one or more levels of cache, such as a level one cache 210 and a level two cache 212, a processor core 214, and registers 216. Example processor cores 214 may include Arithmetic Logic Units (ALUs), Floating Point Units (FPUs), digital signal processing cores (DSP cores), or any combination thereof. The example memory controller 218 may be used with the processor 204, or in some implementations the memory controller 218 may be an internal part of the processor 204.
Depending on the desired configuration, system memory 206 may be any type of memory, including but not limited to: volatile memory (such as RAM), non-volatile memory (such as ROM, flash memory, etc.), or any combination thereof. The physical memory in the computing device is usually referred to as a volatile memory RAM, and data in the disk needs to be loaded into the physical memory to be read by the processor 204. System memory 206 may include an operating system 220, one or more applications 222, and program data 224. In some implementations, the application 222 can be arranged to execute instructions on the operating system with the program data 224 by the one or more processors 204. Operating system 220 may be, for example, Linux, Windows, or the like, which includes program instructions for handling basic system services and for performing hardware-dependent tasks. The application 222 includes program instructions for implementing various user-desired functions, and the application 222 may be, for example, but not limited to, a browser, instant messenger, a software development tool (e.g., an integrated development environment IDE, a compiler, etc.), and the like. When the application 222 is installed into the computing device 200, a driver module may be added to the operating system 220.
When the computing device 200 is started, the processor 204 reads program instructions of the operating system 220 from the memory 206 and executes them. Applications 222 run on top of operating system 220, utilizing the interface provided by operating system 220 and the underlying hardware to implement various user-desired functions. When the user starts the application 222, the application 222 is loaded into the memory 206, and the processor 204 reads the program instructions of the application 222 from the memory 206 and executes the program instructions.
Computing device 200 also includes storage device 232, storage device 232 including removable storage 236 and non-removable storage 238, each of removable storage 236 and non-removable storage 238 being connected to storage interface bus 234.
Computing device 200 may also include an interface bus 240 that facilitates communication from various interface devices (e.g., output devices 242, peripheral interfaces 244, and communication devices 246) to the basic configuration 202 via the bus/interface controller 230. The example output device 242 includes a graphics processing unit 248 and an audio processing unit 250. They may be configured to facilitate communication with various external devices, such as a display or speakers, via one or more a/V ports 252. Example peripheral interfaces 244 can include a serial interface controller 254 and a parallel interface controller 256, which can be configured to facilitate communications with external devices such as input devices (e.g., keyboard, mouse, pen, voice input device, touch input device) or other peripherals (e.g., printer, scanner, etc.) via one or more I/O ports 258. An example communication device 246 may include a network controller 260, which may be arranged to facilitate communications with one or more other computing devices 262 over a network communication link via one or more communication ports 264.
A network communication link may be one example of a communication medium. Communication media may typically be embodied by computer readable instructions, data structures, program modules, and may include any information delivery media, such as carrier waves or other transport mechanisms, in a modulated data signal. A "modulated data signal" may be a signal that has one or more of its data set or its changes made in such a manner as to encode information in the signal. By way of non-limiting example, communication media may include wired media such as a wired network or private-wired network, and various wireless media such as acoustic, Radio Frequency (RF), microwave, Infrared (IR), or other wireless media. The term computer readable media as used herein may include both storage media and communication media.
In a computing device 200 according to the invention, the application 222 includes instructions for performing the data processing method 300 of the invention, which may instruct the processor 204 to perform the coefficient data processing method 300 of the invention.
FIG. 3 illustrates a flow diagram of a data processing method 300, the method 300 being suitable for execution in a data processing system (such as the data processing system shown in FIG. 1), according to one embodiment of the invention.
As shown in fig. 3, the method 300 starts at step S310, and the client initiates a data request according to the business requirement, where the data request includes, but is not limited to, data query, database creation, data update, and the like. The data request comprises target data to be inquired and a target database where the target data is located. According to an embodiment of the present invention, in order for the client a to access the test _ table1 table in the TestDB1 database according to the hive data warehouse architecture, the database name TestDB1 (target database) and the table name test _ table1 (target data) are written into the corresponding data query request.
Before the client sends the data request to the proxy router, the identity authentication with the proxy router is completed through step S320. In step S320, the client completes authentication with the proxy router based on kerberos.
According to one embodiment of the present invention, a plurality of proxy routers are included in the data processing system, and the list of proxy routers in client a is shown in table1, which contains the proxy routers and their address information available in the current data processing system. In order to achieve load balancing among the proxy routers, when selecting a proxy router, the client may select a rule, such as polling or random selection, through a preset rule (i.e., a first preset rule).
TABLE1
| Proxy router | IP address |
| Proxy router 1 | IP_Address1 |
| Proxy router 2 | IP_Address2 |
| Proxy router 3 | IP_Address3 |
According to an embodiment of the present invention, the client needs to complete authentication with the proxy router, in this embodiment, the client a needs to complete authentication with the proxy router 2 based on kerberos when first connecting, and after the authentication passes, the client a will apply for a proxy token from the proxy router 2, and then the proxy token is used for communication authentication between the client a and the proxy router 2. The proxy router 2 stores the token information distributed for the client A locally, meanwhile, the proxy router 2 stores the token information in a zookeeper, all proxy routers and database connectors are connected to the same zookeeper to share the existing token information, and therefore the token applied by the client A to the proxy router 2 can be used for finishing authentication between the client A and the proxy router and can also be used for finishing authentication when the proxy router 2 carrying user identity information of the client A communicates with a target database manager. Specific details regarding the kerberos mechanism are well known to those skilled in the art and will not be described in detail herein.
The process then proceeds to step S330, where the data request is sent to the target proxy router via the shared key obtained in step S320. In this embodiment, the client a sends the data query request to the proxy router 2.
Proceeding to step S340, the proxy router sends the data request to the database manager. According to the data processing method, all the agent routers are connected with the same routing information database, and routing information is managed uniformly through the routing information database, so that the routing information is maintained conveniently.
According to one embodiment of the invention, the routing information database includes fields for: database manager names (MetaName), database managers (URIs), databases (MappingDBs), newly created database identifiers (IsNewDBTo), description information (Desc), etc., and table 2 shows a representation of a routing information database of an embodiment.
TABLE 2
In this step, the proxy router needs to first parse the target database from the data request, and query the database manager corresponding to the target database from the routing information table according to the database name. Continuing with the data request "access to the test _ table1 table in the TestDB1 database" as an example, the proxy router parses the database name "TestDB 1" from the data request, and queries the routing information database (table 2) for the database manager corresponding to the TestDB 1. One database manager is usually implemented by multiple MetaStore instances to achieve load balancing and speed up data queries. In this embodiment, the data management corresponding to TestDB1 includes two metastores, which are MetaStore1 and MetaStore2, respectively, IP addresses are MetaStore1_ Address and MetaStore2_ Address, respectively, and the selection result of the agent router according to the preset rule (i.e., the second preset rule) is MetaStore2, the data request is sent to the MetaStore, and before sending, the agent router also needs to complete the identity authentication with MetaStore based on kerberos. The invention does not limit the selection rule of the MetaStore.
According to another embodiment of the invention, if the proxy router analyzes the name of the database from the data request and does not acquire the mapping relation between the database and the database manager, the data query is failed, and the message of the query failure is directly returned.
According to yet another embodiment of the present invention, the IsNewDBTo field in the routing information database (e.g., table 2) is typically used to identify a newly added database manager that is relatively low loaded, and when the data request is to create a database, the IsNewDBTo field is selected as the true database manager, which creates a new database in the database storage to which the database manager is connected, and stores the newly created database in the routing information database in association with the database manager.
And then, the process goes to step S350, where the database manager obtains the target data from the corresponding database according to the data request, and returns the target data to the client through the proxy router. In the above embodiment, the selected MetaStore obtains the corresponding target data from the database, for example, the MetaStore2 obtains the content of table test _ table1 from the database TestDB1 and returns the content to the proxy router, and the proxy router returns the content to the client a.
According to the data processing method of the invention, by adding the proxy router in the data processing system, the client firstly sends the data request to the proxy router, the proxy router sends the data request to the corresponding database manager according to the data request information, the database manager completes the corresponding operations of data query, database creation and the like according to the data request, and returns response information, different data managers are connected with different databases, a plurality of databases are connected through a plurality of database managers, therefore, the transverse expansion of the database layer is realized, the system performance bottleneck caused by the rapid increase of the service data is eliminated, the data query time can be shortened, the execution efficiency of the service operation is improved, the system performance bottleneck caused by the increase of the source data is eliminated, the system stability is improved, and the potential safety hazard in the system operation process is eliminated.
Furthermore, the agent routers share the same routing information database, so that all agent routes are Kerberos, access users can be authenticated, and malicious users can be prevented from tampering stored data.
The various techniques described herein may be implemented in connection with hardware or software or, alternatively, with a combination of both. Thus, the methods and apparatus of the present invention, or certain aspects or portions thereof, may take the form of program code (i.e., instructions) embodied in tangible media, such as removable hard drives, U.S. disks, floppy disks, CD-ROMs, or any other machine-readable storage medium, wherein, when the program is loaded into and executed by a machine, such as a computer, the machine becomes an apparatus for practicing the invention.
In the case of program code execution on programmable computers, the computing device will generally include a processor, a storage medium readable by the processor (including volatile and non-volatile memory and/or storage elements), at least one input device, and at least one output device. Wherein the memory is configured to store program code; the processor is configured to execute the data processing method of the present invention according to instructions in the program code stored in the memory.
By way of example, and not limitation, readable media may comprise readable storage media and communication media. Readable storage media store information such as computer readable instructions, data structures, program modules or other data. Communication media typically embodies computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media. Combinations of any of the above are also included within the scope of readable media.
In the description provided herein, algorithms and displays are not inherently related to any particular computer, virtual system, or other apparatus. Various general purpose systems may also be used with examples of this invention. The required structure for constructing such a system will be apparent from the description above. Moreover, the present invention is not directed to any particular programming language. It is appreciated that a variety of programming languages may be used to implement the teachings of the present invention as described herein, and any descriptions of specific languages are provided above to disclose the best mode of the invention.
In the description provided herein, numerous specific details are set forth. It is understood, however, that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures and techniques have not been shown in detail in order not to obscure an understanding of this description.
It should be appreciated that in the foregoing description of exemplary embodiments of the invention, various features of the invention are sometimes grouped together in a single embodiment, figure, or description thereof for the purpose of streamlining the disclosure and aiding in the understanding of one or more of the various inventive aspects. However, the disclosed method should not be interpreted as reflecting an intention that: that the invention as claimed requires more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive aspects lie in less than all features of a single foregoing disclosed embodiment. Thus, the claims following the detailed description are hereby expressly incorporated into this detailed description, with each claim standing on its own as a separate embodiment of this invention.
Those skilled in the art will appreciate that the modules or units or components of the devices in the examples disclosed herein may be arranged in a device as described in this embodiment or alternatively may be located in one or more devices different from the devices in this example. The modules in the foregoing examples may be combined into one module or may be further divided into multiple sub-modules.
Those skilled in the art will appreciate that the modules in the device in an embodiment may be adaptively changed and disposed in one or more devices different from the embodiment. The modules or units or components of the embodiments may be combined into one module or unit or component, and furthermore they may be divided into a plurality of sub-modules or sub-units or sub-components. All of the features disclosed in this specification (including any accompanying claims, abstract and drawings), and all of the processes or elements of any method or apparatus so disclosed, may be combined in any combination, except combinations where at least some of such features and/or processes or elements are mutually exclusive. Each feature disclosed in this specification (including any accompanying claims, abstract and drawings) may be replaced by alternative features serving the same, equivalent or similar purpose, unless expressly stated otherwise.
Furthermore, those skilled in the art will appreciate that while some embodiments described herein include some features included in other embodiments, rather than other features, combinations of features of different embodiments are meant to be within the scope of the invention and form different embodiments. For example, in the following claims, any of the claimed embodiments may be used in any combination.
Furthermore, some of the described embodiments are described herein as a method or combination of method elements that can be performed by a processor of a computer system or by other means of performing the described functions. A processor having the necessary instructions for carrying out the method or method elements thus forms a means for carrying out the method or method elements. Further, the elements of the apparatus embodiments described herein are examples of the following apparatus: the apparatus is used to implement the functions performed by the elements for the purpose of carrying out the invention.
As used herein, unless otherwise specified the use of the ordinal adjectives "first", "second", "third", etc., to describe a common object, merely indicate that different instances of like objects are being referred to, and are not intended to imply that the objects so described must be in a given sequence, either temporally, spatially, in ranking, or in any other manner.
While the invention has been described with respect to a limited number of embodiments, those skilled in the art, having benefit of this description, will appreciate that other embodiments can be devised which do not depart from the scope of the invention as described herein. Furthermore, it should be noted that the language used in the specification has been principally selected for readability and instructional purposes, and may not have been selected to delineate or circumscribe the inventive subject matter. Accordingly, many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the appended claims. The present invention has been disclosed in an illustrative rather than a restrictive sense, and the scope of the present invention is defined by the appended claims.
Claims (10)
1. A data processing method adapted to be executed in a data processing system comprising a client, a proxy router, a database manager, a database, the method comprising:
a client sends a data request to the proxy router, wherein the data request comprises target data and a target database corresponding to the target data;
the proxy router acquires a target database in the data request and acquires a database manager corresponding to the target database according to the target database;
the proxy router sends the data request to a database manager corresponding to the target database;
and the database manager completes data operation in the corresponding database according to the data request and returns a response message.
2. The method of claim 1, wherein the client stores a proxy router list having stored therein proxy routers and address information of the proxy routers, and sending the data request to the proxy router comprises:
and the client selects a target proxy router from the proxy router list according to a first preset rule, and sends the data request to the target proxy router according to the address information of the target proxy router.
3. The method according to claim 1 or 2, wherein the data processing system further comprises a routing information database, the routing information database stores a mapping relationship between a database and a database manager, all the agent routers are in communication connection with the routing information database, and the obtaining the database manager corresponding to the target database according to the target database comprises:
and the agent router queries the routing information database according to the target database and acquires a database manager corresponding to the target database.
4. The method of claim 3, wherein the database manager comprises a plurality of MetaStores, and the proxy router sending the data request to a database manager corresponding to the target database comprises:
selecting a target MetasTore from the plurality of MetasTores according to a second preset rule;
and sending the data request to the target MetaStore.
5. The method of claim 3 or 4, wherein the routing table further comprises a newly created database identification field, and when the data request is to create a new database, the obtaining the database manager corresponding to the target database according to the target database comprises:
a database manager is obtained that identifies the newly created database as genuine.
6. The method of any of claims 1-5, the proxy router supporting kerberos, the method further comprising:
the client completes authentication with the proxy router based on kerberos;
the proxy router completes authentication with the database manager based on kerberos.
7. The method of any one of claims 1-6, wherein the database is in a master-slave configuration, comprising a master database and a slave database, the master database and slave database being data-consistent.
8. A data processing system comprises a client, a proxy router, a database manager and a database, wherein,
the client is suitable for sending a data request to the proxy router, wherein the data request comprises target data and a target database corresponding to the target data;
the agent router is suitable for acquiring a target database in the data request and acquiring a database manager corresponding to the target database according to the target database; the data request is further sent to a database manager corresponding to the target database;
the database manager is suitable for finishing data operation in a corresponding database according to the data request and returning a response message;
a database adapted to store target data.
9. A computing device, comprising:
at least one processor; and
a memory storing program instructions, wherein the program instructions are configured to be executed by the at least one processor, the program instructions comprising instructions for performing the method of any of claims 1-7.
10. A readable storage medium storing program instructions that, when read and executed by a computing device, cause the mobile terminal to perform the method of any of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110377822.1A CN112988874A (en) | 2021-04-08 | 2021-04-08 | Data processing method, system, computing device and readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110377822.1A CN112988874A (en) | 2021-04-08 | 2021-04-08 | Data processing method, system, computing device and readable storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112988874A true CN112988874A (en) | 2021-06-18 |
Family
ID=76339445
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110377822.1A Pending CN112988874A (en) | 2021-04-08 | 2021-04-08 | Data processing method, system, computing device and readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112988874A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116112550A (en) * | 2022-12-29 | 2023-05-12 | 上海芯赛云计算科技有限公司 | Data processing method and device, storage medium and electronic device |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP2778962A1 (en) * | 2013-03-14 | 2014-09-17 | Sap Ag | Silo-aware databases |
| US20150150125A1 (en) * | 2013-11-28 | 2015-05-28 | Cyber-Ark Software Ltd. | Correlation based security risk identification |
| CN105426487A (en) * | 2015-11-20 | 2016-03-23 | 北京京东尚科信息技术有限公司 | Distributed database access control method and equipment, distributed database system and expansion method of distributed database system |
| CN107172192A (en) * | 2017-06-19 | 2017-09-15 | 西安工程大学 | SaaS NMS data management frameworks based on mixing storage |
| CN109542611A (en) * | 2017-09-21 | 2019-03-29 | 中国移动通信集团重庆有限公司 | Database, that is, service system, database dispatching method, equipment and storage medium |
| CN111177112A (en) * | 2019-12-06 | 2020-05-19 | 陕西上讯信息技术有限公司 | Database blocking method and device based on operation and maintenance management system and electronic equipment |
| CN111367983A (en) * | 2020-03-10 | 2020-07-03 | 中国联合网络通信集团有限公司 | Database access method, system, device and storage medium |
| CN111666265A (en) * | 2020-06-04 | 2020-09-15 | 南京领行科技股份有限公司 | Data management method, device, server and storage medium |
| CN112364110A (en) * | 2020-11-17 | 2021-02-12 | 深圳前海微众银行股份有限公司 | Metadata management method, device and equipment and computer storage medium |
-
2021
- 2021-04-08 CN CN202110377822.1A patent/CN112988874A/en active Pending
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP2778962A1 (en) * | 2013-03-14 | 2014-09-17 | Sap Ag | Silo-aware databases |
| US20150150125A1 (en) * | 2013-11-28 | 2015-05-28 | Cyber-Ark Software Ltd. | Correlation based security risk identification |
| CN105426487A (en) * | 2015-11-20 | 2016-03-23 | 北京京东尚科信息技术有限公司 | Distributed database access control method and equipment, distributed database system and expansion method of distributed database system |
| CN107172192A (en) * | 2017-06-19 | 2017-09-15 | 西安工程大学 | SaaS NMS data management frameworks based on mixing storage |
| CN109542611A (en) * | 2017-09-21 | 2019-03-29 | 中国移动通信集团重庆有限公司 | Database, that is, service system, database dispatching method, equipment and storage medium |
| CN111177112A (en) * | 2019-12-06 | 2020-05-19 | 陕西上讯信息技术有限公司 | Database blocking method and device based on operation and maintenance management system and electronic equipment |
| CN111367983A (en) * | 2020-03-10 | 2020-07-03 | 中国联合网络通信集团有限公司 | Database access method, system, device and storage medium |
| CN111666265A (en) * | 2020-06-04 | 2020-09-15 | 南京领行科技股份有限公司 | Data management method, device, server and storage medium |
| CN112364110A (en) * | 2020-11-17 | 2021-02-12 | 深圳前海微众银行股份有限公司 | Metadata management method, device and equipment and computer storage medium |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116112550A (en) * | 2022-12-29 | 2023-05-12 | 上海芯赛云计算科技有限公司 | Data processing method and device, storage medium and electronic device |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11836135B1 (en) | Method and system for transparent database query caching | |
| US9996565B2 (en) | Managing an index of a table of a database | |
| US9032017B1 (en) | Method and system for transparent read-write query routing when load balancing databases | |
| WO2020024408A1 (en) | Test device, test method and storage medium | |
| US9483515B2 (en) | Managing a table of a database | |
| EP3376403A1 (en) | Method of accessing distributed database and device providing distributed data service | |
| CN103312624A (en) | Message queue service system and method | |
| US20130085895A1 (en) | High throughput global order promising system | |
| US20190197040A1 (en) | Method and apparatus for processing join query | |
| CN110633378A (en) | Graph database construction method supporting super-large scale relational network | |
| WO2017148297A1 (en) | Method and device for joining tables | |
| US20220300477A1 (en) | Data Read/Write Method and Apparatus for Database | |
| US8930518B2 (en) | Processing of write requests in application server clusters | |
| US11409781B1 (en) | Direct storage loading for adding data to a database | |
| WO2016101759A1 (en) | Data routing method, data management device and distributed storage system | |
| CN112988874A (en) | Data processing method, system, computing device and readable storage medium | |
| CN111400301A (en) | Data query method, device and equipment | |
| JP6189266B2 (en) | Data processing apparatus, data processing method, and data processing program | |
| CN113704295A (en) | Service request processing method and system and electronic equipment | |
| US9537941B2 (en) | Method and system for verifying quality of server | |
| CN110417777A (en) | A kind of method and device communicated between micro services of optimization | |
| US11620285B2 (en) | Automatic database query translation | |
| Carstoiu et al. | High performance eventually consistent distributed database Zatara | |
| CN111680069B (en) | Database access method and device | |
| WO2022032532A1 (en) | Sharding for workflow applications in serverless architectures |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20210618 |