CN115481187B - Data reading and writing method, device and storage medium - Google Patents

Data reading and writing method, device and storage medium Download PDF

Info

Publication number
CN115481187B
CN115481187B CN202211191106.5A CN202211191106A CN115481187B CN 115481187 B CN115481187 B CN 115481187B CN 202211191106 A CN202211191106 A CN 202211191106A CN 115481187 B CN115481187 B CN 115481187B
Authority
CN
China
Prior art keywords
database
data
target data
access
edge
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202211191106.5A
Other languages
Chinese (zh)
Other versions
CN115481187A (en
Inventor
林伟
李贝贝
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing Wellintech Co Ltd
Original Assignee
Beijing Wellintech Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing Wellintech Co Ltd filed Critical Beijing Wellintech Co Ltd
Priority to CN202211191106.5A priority Critical patent/CN115481187B/en
Publication of CN115481187A publication Critical patent/CN115481187A/en
Application granted granted Critical
Publication of CN115481187B publication Critical patent/CN115481187B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/25Integrating or interfacing systems involving database management systems
    • G06F16/252Integrating or interfacing systems involving database management systems between a Database Management System and a front-end application
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/24Querying
    • G06F16/242Query formulation
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/24Querying
    • G06F16/245Query processing
    • G06F16/2453Query optimisation
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/25Integrating or interfacing systems involving database management systems
    • G06F16/254Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses

Abstract

The application relates to the technical field of data storage, and discloses a data reading and writing method, a device and a storage medium, wherein the method comprises the following steps: and searching target data corresponding to the data reading request in the terminal database and the remote database in response to the received data reading request, wherein the terminal database and the remote database are respectively different in corresponding region information, are connected through a network, and return the target data to a client for sending the data reading request, and the target data is stored in the terminal database or the remote database based on the attribute characteristics of the target data corresponding to the data writing request after receiving the data writing request, so that the distributed storage of the data is realized, the condition of data loss caused by wide data distribution region is avoided, and the efficiency and the accuracy of data query are improved.

Description

Data reading and writing method, device and storage medium
Technical Field
The application relates to the technical field of data storage, and provides a data reading and writing method, a data reading and writing device and a storage medium.
Background
At present, the database structure is relatively simple, and in the storage process, data is stored in the database in the server, so that different clients in different regions can access the stored data at the same time.
However, frequent access to the database by the client may reduce the query performance of the server, that is, a large amount of communication resources are consumed when querying big data, especially when the clients located in different regions query the same data at the same time, the problems of data loss and poor query performance may occur.
Disclosure of Invention
The embodiment of the application provides a data reading and writing method, a data reading and writing device and a storage medium, which are used for improving the efficiency and the accuracy of data query.
The specific technical scheme provided by the application is as follows:
in a first aspect, an embodiment of the present application provides a data reading and writing method, including:
searching target data corresponding to the data reading request in the terminal database and the remote database in response to the received data reading request, wherein the regional information corresponding to each of the terminal database and the remote database is different, and the terminal database and the remote database are connected through a network;
returning the target data to the client side sending the data reading request;
the target data is stored in the terminal database or the remote database based on the attribute characteristics of the target data corresponding to the data writing request after the data writing request is received.
Optionally, returning the target data to the client that sent the data read request includes:
if the target data is found in the terminal database, the target data is directly returned to the client for sending the data reading request;
if the target data is found in the remote database, synchronizing the target data to the terminal database, and returning the target data to the client for sending the data reading request by using the terminal database.
Optionally, the end database is at the same physical node as the client of the data read request.
Optionally, if the remote databases include a cloud database cluster and at least one edge database cluster, connecting at least two remote databases by:
connecting any two cloud data nodes through a network, respectively connecting all the edge data nodes in each edge database cluster through the network, and respectively connecting each cloud data node with at least one edge data node through the network;
the cloud data clusters are at least one remote database selected from at least two remote databases based on a preset rule, the edge database clusters are obtained by dividing the remote databases except for cloud data nodes in the at least two remote databases based on regional information corresponding to each remote database, the edge data nodes are each remote database in the edge database clusters, and the number of the edge database clusters is at least one;
And connecting the cloud data nodes and/or the edge data nodes corresponding to the region information of the end database with the end database through a network.
Optionally, returning the target data to the client that sent the data read request includes:
determining a storage database storing target data, wherein the storage database is any one of an end database, an edge data node and a cloud data node;
determining an access database connected with a client side sending a data reading request, wherein the access database and the client side sending the data reading request are positioned at the same physical node;
if the access database sending the data reading request and the storage database are the same database, returning the target data from the storage database to the client sending the data reading request;
if the access database sending the data reading request is not the same database as the storage database, synchronizing the target data to the access database through the storage database, and returning the target data to the client sending the data reading request by using the access database.
Optionally, if the access database sending the data reading request is not the same database as the storage database, synchronizing the target data to the access database through the storage database, and returning the target data to the client sending the data reading request by using the access database, including:
If the access database is an edge data node, the access database and the storage database are not the same database, and the access database and the storage database belong to the same edge database cluster, the target data is sent to the access database from the storage database, and the target data is returned to a client for sending a data reading request through the access database;
if the access database is an edge data node, the access database and the storage database are not the same database, and the access database and the storage database do not belong to the same edge database cluster, then the target data is sent to the cloud data node from the storage database, the target data is returned to the edge database cluster to which the access database belongs through the cloud data node, and the target data is sent to a client of a data reading request through the access database in the edge database cluster to which the access database belongs;
if the access database is a cloud data node and the access database and the storage database are not the same database, sending the target data from the storage database to the cloud data node, and sending the target data to a client of a data reading request through the access database;
if the access database is an end database and the access database and the storage database are not the same database, sending the target data from the storage database to the cloud data node, returning the target data to an edge database cluster connected with the access database through the cloud data node, returning the target data to the access database through the edge database cluster connected with the access database, and sending the target data to a client of a data reading request through the access database.
Optionally, the target data is stored in the end database or the remote database by:
responding to the received data writing request, and determining attribute characteristics of target data corresponding to the data writing request;
determining an end database or a far-end database matched with the attribute characteristics based on a preset storage rule;
writing the target data into the matched terminal database or the remote database, writing the identification generated based on the target data into the matched terminal database or the remote database, and sending the identification to the client, wherein the identification is a global variable.
Optionally, if the network link between the access database and the storage database fails, after responding to the received data writing request, the method further includes:
storing target data corresponding to the data writing request into an access database;
if the network link fault between the access database and the storage database is recovered to be normal, the method further comprises:
triggering the access database to store the target data in the storage database based on the attribute characteristics of the target data.
Optionally, in response to the received data reading request, searching the terminal database and the remote database for target data corresponding to the data reading request, including:
In response to the received data reading request, determining an identifier corresponding to the data reading request from a client side sending the data reading request;
searching the identifiers stored in the database, which are the same as the identifiers corresponding to the data reading requests, in the end database and the far-end database based on the identifiers corresponding to the data reading requests;
target data corresponding to the identification stored in the database is determined as target data corresponding to the data read request.
In a second aspect, an embodiment of the present application further provides a data reading and writing device, including:
the searching unit is used for responding to the received data reading request and searching target data corresponding to the data reading request in the terminal database and the remote database, wherein the terminal database and the remote database are respectively different in corresponding regional information and are connected through a network;
the return unit is used for returning the target data to the client side sending the data reading request;
the target data is stored in the terminal database or the remote database based on the attribute characteristics of the target data corresponding to the data writing request after the data writing request is received.
Optionally, if the remote databases include a cloud database cluster and at least one edge database cluster, connecting at least two remote databases by:
Connecting any two cloud data nodes through a network, respectively connecting all the edge data nodes in each edge database cluster through the network, and respectively connecting each cloud data node with at least one edge data node through the network;
the cloud data clusters are at least one remote database selected from at least two remote databases based on preset rules, the edge database clusters are obtained by dividing the remote databases except for cloud data nodes in the at least two remote databases based on regional information corresponding to the remote databases, the edge data nodes are the remote databases in the edge database clusters, and the number of the edge database clusters is at least one.
Optionally, the target data is returned to the client sending the data reading request, and the return unit is configured to:
determining a storage database storing target data, wherein the storage database is any one of an end database, an edge data node and a cloud data node;
determining an access database connected with a client side sending a data reading request, wherein the access database and the client side sending the data reading request are positioned at the same physical node;
If the access database sending the data reading request and the storage database are the same database, returning the target data from the storage database to the client sending the data reading request;
if the access database sending the data reading request is not the same database as the storage database, synchronizing the target data to the access database through the storage database, and returning the target data to the client sending the data reading request by using the access database.
Optionally, if the access database sending the data read request is not the same database as the storage database, synchronizing the target data to the access database through the storage database, and returning the target data to the client sending the data read request by using the access database, wherein the returning unit is configured to:
if the access database is an edge data node, the access database and the storage database are not the same database, and the access database and the storage database belong to the same edge database cluster, the target data is sent to the access database from the storage database, and the target data is returned to a client for sending a data reading request through the access database;
if the access database is an edge data node, the access database and the storage database are not the same database, and the access database and the storage database do not belong to the same edge database cluster, then the target data is sent to the cloud data node from the storage database, the target data is returned to the edge database cluster to which the access database belongs through the cloud data node, and the target data is sent to a client of a data reading request through the access database in the edge database cluster to which the access database belongs;
If the access database is a cloud data node and the access database and the storage database are not the same database, sending the target data from the storage database to the cloud data node, and sending the target data to a client of a data reading request through the access database;
if the access database is an end database and the access database and the storage database are not the same database, sending the target data from the storage database to the cloud data node, returning the target data to an edge database cluster connected with the access database through the cloud data node, returning the target data to the access database through the edge database cluster connected with the access database, and sending the target data to a client of a data reading request through the access database.
Optionally, the target data is stored in the end database or the remote database by:
responding to the received data writing request, and determining attribute characteristics of target data corresponding to the data writing request;
determining an end database or a far-end database matched with the attribute characteristics based on a preset storage rule;
writing the target data into the matched terminal database or the remote database, writing the identification generated based on the target data into the matched terminal database or the remote database, and sending the identification to the client, wherein the identification is a global variable.
Optionally, if the network link between the access database and the storage database fails, after responding to the received data writing request, the method further includes:
reading and writing target data corresponding to the data writing request into an access database;
if the network link fault between the access database and the storage database is recovered to be normal, the method further comprises:
triggering the access database to read and write the target data into the storage database based on the attribute characteristics of the target data.
Optionally, in response to the received data reading request, searching target data corresponding to the data reading request in the end database and the far-end database, and the searching unit is used for:
in response to the received data reading request, determining an identifier corresponding to the data reading request from a client side sending the data reading request;
searching the identifiers stored in the database, which are the same as the identifiers corresponding to the data reading requests, in the end database and the far-end database based on the identifiers corresponding to the data reading requests;
target data corresponding to the identification stored in the database is determined as target data corresponding to the data read request.
In a third aspect, an intelligent terminal includes:
A memory for storing executable instructions;
a processor for reading and executing executable instructions stored in a memory to implement the method of any one of the first aspects.
In a fourth aspect, a computer readable storage medium, which when executed by a processor, causes the processor to perform the method of any of the first aspects.
The application has the following beneficial effects:
in summary, in the embodiment of the present application, a data reading and writing method, a device and a storage medium are provided, where the method includes: and searching target data corresponding to the data reading request in the terminal database and the remote database in response to the received data reading request, wherein the terminal database and the remote database are respectively different in corresponding region information, are connected through a network, and return the target data to a client for sending the data reading request, and the target data is stored in the terminal database or the remote database based on the attribute characteristics of the target data corresponding to the data writing request after receiving the data writing request, so that the distributed storage of the data is realized, the condition of data loss caused by wide data distribution region is avoided, and the efficiency and the accuracy of data query are improved.
Additional features and advantages of the application will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by practice of the application. The objectives and other advantages of the application will be realized and attained by the structure particularly pointed out in the written description and claims thereof as well as the appended drawings.
Drawings
The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this specification, illustrate embodiments of the application and together with the description serve to explain the application and do not constitute a limitation on the application. In the drawings:
fig. 1 is a schematic diagram of a system architecture of a database system in which a cloud database cluster is connected to an end database according to an embodiment of the present application;
FIG. 2 is a schematic diagram of a system architecture of a database system in which a cloud database cluster and an edge database cluster are connected to an end database according to an embodiment of the present application;
FIG. 3 is a schematic diagram illustrating a cloud database cluster according to an embodiment of the present application;
FIG. 4 is a schematic flow chart of a data reading and writing method according to an embodiment of the application;
FIG. 5 is a flow chart of storing data in combination with a network situation according to an embodiment of the present application;
FIG. 6 is a flowchart of a data reading method according to an embodiment of the present application;
FIG. 7 is a flow chart of searching target data based on a data read request according to an embodiment of the present application;
FIG. 8 is a flowchart illustrating a process of returning target data to a client according to an embodiment of the present application;
FIG. 9 is a flow chart of returning target data based on different storage databases according to an embodiment of the present application;
FIG. 10 is a schematic diagram of a logic architecture of a data read/write device according to an embodiment of the present application;
fig. 11 is a schematic diagram of an entity architecture of the intelligent terminal according to an embodiment of the present application.
Detailed Description
For the purpose of making the objects, technical solutions and advantages of the embodiments of the present application more apparent, the technical solutions of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application, and it is apparent that the described embodiments are some embodiments of the technical solutions of the present application, but not all embodiments. All other embodiments, based on the embodiments described in the present document, which can be obtained by a person skilled in the art without any creative effort, are within the scope of protection of the technical solutions of the present application.
The terms first, second and the like in the description and in the claims and in the above-described figures, are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged where appropriate such that the embodiments of the application described herein may be capable of being practiced otherwise than as specifically illustrated and described.
The preferred embodiments of the present application will be described in detail with reference to the accompanying drawings.
First, the architecture of the database in the present application is described below, and referring to fig. 1, the architecture specifically includes an end database and a cloud database cluster, that is, the far-end database includes only one cloud database cluster. The terminal databases and the cloud database clusters are connected through a network, and the terminal databases are not connected. Referring to fig. 2, when the remote database includes a cloud database cluster and at least one side database cluster, each side database may be connected to the cloud database cluster through a network, and simultaneously, each side database may also be connected to the side database cluster through a network, where each side database cluster is not connected to each other. Referring to fig. 3, the cloud database cluster may include a plurality of cloud data nodes connected through a network, and similarly, the edge database cluster may include a plurality of edge data nodes connected through a network.
The number of the terminal databases and the remote databases in the application is a plurality, and the regional information of each terminal database and each remote database is different, so that the client side positioned in different regional information can store and read data conveniently.
In addition, in order to facilitate the storage and reading of data, the data are synchronized among the remote databases, i.e. when a client a connected to the remote database a wants to store data M in the remote database B, the client a sends the data M to the remote database a first, and then the remote database a sends the data M to the remote database B.
It should be further noted that, in order to effectively save storage space, each data may be stored in only one of the above-mentioned respective end databases or the remote databases, and data transmission between the respective end databases and/or the remote databases may be performed without backing up in the other database.
In addition, the data stored in the end database and the remote database may be real-time data, historical data, etc., that is, the time limitation of the data stored in each end database and the remote database is not limited.
The connection relationship between the databases is specifically described below, and if the remote databases include a cloud database cluster and at least one edge database cluster, at least two remote databases are connected in the following manner:
firstly, it is to be noted that the cloud data cluster is at least one remote database selected from at least two remote databases based on a preset rule, the edge database cluster is obtained by dividing remote databases except for cloud data nodes in the at least two remote databases based on regional information corresponding to each remote database, the edge data nodes are each remote database in the edge database cluster, and the number of groups of the edge database clusters is at least one group.
That is, when the remote database includes a cloud database cluster and at least one edge database cluster, at least one remote database is selected from the cloud database clusters as a cloud data node, and specific selection rules may be preset, including but not limited to, large storage space, medium location of regional information, and the like. After at least one cloud data node is selected, the selected cloud data nodes are automatically classified into a group, namely a cloud database cluster, and all cloud data nodes in the cloud database cluster are connected through a network.
And dividing the rest of the remote databases, namely the remote databases except for the cloud data nodes in at least two remote databases, to obtain at least one group of edge database clusters, wherein the division is based on the fact that the regional information is generally closer and is classified into one group. Any one of the remote databases in each of the edge database clusters is called an edge data node, and the edge database clusters are connected with the cloud data node through a network.
Specifically, any two cloud data nodes are connected through a network, each side data node in each side database cluster is connected through the network, and each cloud data node is connected with at least one side data node through the network.
In an optional implementation manner, cloud data nodes and/or edge data nodes corresponding to the region information of the end database are connected with the end database through a network.
In another alternative embodiment, if the number of the end databases is at least two, the connection relationship between the at least two end databases is determined by:
it should be noted that, the end databases are usually databases closest to the user, the region information corresponding to each end database is usually different, and when the number of the end databases is at least two, the following operations are performed for each end database:
1) And determining the region information with the minimum difference value between the region information corresponding to the terminal database from the region information corresponding to each cloud data node and each side data node.
In order to establish a connection between the end database and the far-end database, that is, establish a connection between the end database and the edge data nodes, it is necessary to determine a difference value between the region information corresponding to each edge data node and the region information corresponding to the end database, and determine a cloud data node and/or an edge data node corresponding to the region information with the smallest difference value from the difference value, that is, determine the edge data node closest to one end database.
2) And connecting the cloud data nodes and/or the edge data nodes corresponding to the region information with the minimum difference with the end database through a network.
In the implementation process, after the edge data node closest to the edge data node is determined, the edge data node is connected with the corresponding end database through a network.
Continuing to describe the manner in which the data is written into the database, the target data is stored in the end database or the remote database in the following manner, as shown in fig. 4:
step 101: and responding to the received data writing request, and determining the attribute characteristics of the target data corresponding to the data writing request.
In the implementation process, when a client wants to write data into a certain database, the client sends a data writing request, and after receiving the data writing request, a database system responds to the data writing request to determine the attribute characteristics of target data corresponding to the data writing request, wherein the attribute characteristics include but are not limited to the use frequency, importance degree, data size and the like.
The target data is stored in the end database or the remote database based on the attribute characteristics of the target data corresponding to the data writing request after the data writing request is received.
Since the network connection inside the database system may fail, if the network link between the access database and the storage database fails, after responding to the received data writing request, referring to fig. 5, the method further includes:
the access database is a database directly connected with the client side sending the data writing request, and the end database, the edge data node and the cloud data node can be externally connected with the client side, so the access database can be any one of the end database, the edge data node and the cloud data node. The storage database is a database in which data corresponding to the data writing request is actually written, and the storage database can be any one of an end database, an edge data node and a cloud data node.
Step 1011: and storing target data corresponding to the data writing request into an access database.
In the implementation process, when a network link between an access database and a storage database fails, target data corresponding to a data writing request is stored in the access database, namely, target data which a client wants to store is stored in a database directly connected with the client (namely, the access database), and the target data is temporarily stored by the access database.
Step 1012: and if the network link between the access database and the storage database fails, the network link is recovered to be normal.
In the implementation process, after the network link fault between the access database and the storage database is determined to be recovered to be normal, namely, the network link between the access database and the storage database of the target data is normal.
Step 1013: triggering the access database to store the target data in the storage database based on the attribute characteristics of the target data.
Further, the access database is triggered to store the target data into the storage database by the access database for temporarily storing the target data after determining the attribute characteristics of the target data, so that the final storage purpose of the client side for sending the data writing request is realized.
Step 102: and determining an end database or a far-end database matched with the attribute features based on a preset storage rule.
In the implementation process, after determining the attribute characteristics of the target data, the attribute characteristics are matched with a preset storage rule, and the storage rule is assumed to be: when determining that the attribute features are high in importance, one of a matched database, namely an end database or a far-end database (an edge data node or a cloud data node), needs to be found.
Step 103: writing the target data into the matched terminal database or the remote database, writing the identification generated based on the target data into the matched terminal database or the remote database, and sending the identification to the client, wherein the identification is a global variable.
Further, the target data is written into the matched end database or the far-end database, and in order to uniquely identify the written data, in the embodiment of the application, the identifier is generated according to the target data, and the identifier is a global variable, so that the identifier is stored in the matched end database or the far-end database while the target data is stored.
In order to facilitate the client to find the stored target data, the identifier is also sent to the client, and in order to ensure the uniqueness of the target data, the identifier is set as a global variable.
The following describes a data reading process based on the database system of the present application, referring to fig. 6, a data reading method includes:
step 201: and searching target data corresponding to the data reading request in the terminal database and the remote database in response to the received data reading request.
In the data reading process, a client side firstly sends a data reading request, and a database system searches corresponding target data according to the data reading request after receiving the data reading request.
Referring to fig. 7, the method specifically includes:
step 2011: in response to the received data read request, an identification corresponding to the data read request is determined from the client that sent the data read request.
In the data storage process, the target data and the identifiers are in one-to-one correspondence, and the identifiers corresponding to the stored target data are prestored in each client, so that in the data reading process, in order to conveniently and rapidly find the target data, the identifiers are taken as the searching basis, specifically, after the data reading request is received, the identifier corresponding to the data reading request is determined from the client sending the data reading request, and the identifier characterizes the target data to be read.
Step 2012: and searching the identifiers stored in the database, which are the same as the identifiers corresponding to the data reading requests, in the end database and the far-end database based on the identifiers corresponding to the data reading requests.
In the implementation process, after the identification corresponding to the data reading request is determined, namely, after the target data to be read is determined, the identifications stored in the databases, which are the same as the identification corresponding to the data reading request, are searched in each end database and the remote database, namely, the stored identifications are searched in all databases.
Step 2013: target data corresponding to the identification stored in the database is determined as target data corresponding to the data read request.
In the implementation process, after the stored identifier is found in the database, the target data corresponding to the identifier is determined to be the target data corresponding to the data reading request, and the target data can be returned to the client.
Step 202: and returning the target data to the client side sending the data reading request.
In the implementation process, the target data is returned to the client for sending the data reading request, which comprises the following two cases:
first case: if the target data is found in the end database, the target data is directly returned to the client for sending the data reading request.
For example, the target data a is found out of all the data stored in the end database a, and the found target data a is directly returned to the client 1 that sends the data reading request.
Second case: if the target data is found in the remote database, synchronizing the target data to the terminal database, and returning the target data to the client for sending the data reading request by using the terminal database.
For example, the target data B is found out of all the data stored in the edge database B, the found target data B is synchronized to the end database c, and the end database c is used to return the target data B to the client 2 that sends the data reading request.
It should be noted that in both cases, the end database is at the same physical node as the client of the data read request.
In the implementation process, considering that the region information where the storage database and the access database are located may be different, the method for returning the target data to the client side sending the data reading request, as shown in fig. 8, specifically includes:
step 2021: and determining a storage database storing target data, wherein the storage database is any one of an end database, an edge data node and a cloud data node.
In the implementation process, after the target data is found according to the identification, the database storing the target data is determined as the storage database. Obviously, the storage database is any one of an end database, an edge data node and a cloud data node.
Step 2022: an access database is determined that is connected to a client that sent the data read request, wherein the access database is at the same physical node as the client that sent the data read request.
To determine the return path of the target data, the location of the client is further determined, i.e. the database connected to the client sending the data read request is determined as the access database, i.e. the access database is at the same physical node as the client sending the data read request. Obviously, the access database is any one of an end database, an edge data node and a cloud data node.
It is contemplated that the access database and the storage database may be the same database or may be different databases, and the implementation process is described in the following steps.
Step 2023: if the access database sending the data reading request and the storage database are the same database, the target data is returned from the storage database to the client sending the data reading request.
In the implementation process, if the access database sending the data reading request and the storage database are determined to be the same database, namely, the client side sending the data reading request is connected with the same database, the searched target data is returned to the client side sending the data reading request from the same database.
Step 2024: if the access database sending the data reading request is not the same database as the storage database, synchronizing the target data to the access database through the storage database, and returning the target data to the client sending the data reading request by using the access database.
In the implementation process, if it is determined that the access database sending the data reading request and the storage database are not the same database, that is, the client sending the data reading request is connected with one database, the database storing the target data is another database, the target data needs to be transferred to the client, and the transferred path is the database connected with the storage database.
Specifically, if the access database sending the data read request is not the same database as the storage database, the storage database synchronizes the target data to the access database, and the access database is used to return the target data to the client sending the data read request, as shown in fig. 9, including:
step 20241: if the access database is an edge data node, the access database and the storage database are not the same database, and the access database and the storage database belong to the same edge database cluster, the target data is sent to the access database from the storage database, and the target data is returned to the client for sending the data reading request through the access database.
First case: the client side sending the data reading request is connected with the edge data node, the database storing the target data is another edge data node in the same edge database cluster, and because the edge data nodes in the same edge database cluster are connected through a network, the target data is sent from the storage database (namely one edge data node) to the access database (namely the other edge data node), the storage database sends the target data to the access database, and the target data is returned to the client side sending the data reading request through the access database.
Step 20242: if the access database is an edge data node, the access database and the storage database are not the same database, and the access database and the storage database do not belong to the same edge database cluster, then the target data is sent to the cloud data node from the storage database, the target data is returned to the edge database cluster to which the access database belongs through the cloud data node, and the target data is sent to the client of the data reading request through the access database in the edge database cluster to which the access database belongs.
Second case: the client side sending the data reading request is connected with the edge data node, the database storing the target data is another edge data node in the edge database clusters of different groups, and because the edge data nodes in the edge database clusters of different groups are connectionless, the target data is sent to the cloud data node from the storage database (namely one edge data node) and returned to the edge database cluster to which the access database belongs through the cloud data node, the access database in the edge database cluster receives the target data sent by the cloud data node, and the target data is returned to the client side sending the data reading request through the access database.
Step 20243: if the access database is a cloud data node and the access database and the storage database are not the same database, the target data is sent to the cloud data node from the storage database, and the target data is sent to the client of the data reading request through the access database.
Third case: the client side sending the data reading request is connected with one cloud data node, and the database storing the target data is another database (any one of the end database, the side data node and the cloud data node different from the cloud data node). In this case, the target data is transmitted from the storage database to the connected cloud data nodes, and the cloud data nodes connected to the storage database transmit the target data to the access database (i.e., one of the cloud data nodes), for example, when the storage database is another cloud data node, the other cloud data node transmits the target data to the one of the cloud data nodes, and then the access database returns the target data to the client that transmits the data read request.
Step 20244: if the access database is an end database and the access database and the storage database are not the same database, sending the target data from the storage database to the cloud data node, returning the target data to an edge database cluster connected with the access database through the cloud data node, returning the target data to the access database through the edge database cluster connected with the access database, and sending the target data to a client of a data reading request through the access database.
Fourth case: the client side sending the data reading request is connected with the end database, the database storing the target data is another database (any one of the end database, the side data node and the cloud data node), and because the end databases are connected through the cloud data node, the target data is sent to the cloud data node from the storage database, the cloud data node sends the target data to the side database cluster connected with the access database, the side data node connected with the access database in the side database cluster returns the target data to the access database, and the access database returns the target data to the client side sending the data reading request.
Based on the same inventive concept, referring to fig. 10, an embodiment of the present application provides a data read/write device, including:
the searching unit 1001 is configured to respond to the received data reading request, and search target data corresponding to the data reading request in the end database and the far-end database, where the regional information corresponding to each of the end database and the far-end database is different, and the end database and the far-end database are connected through a network;
A returning unit 1002, configured to return the target data to the client that sends the data reading request;
the target data is stored in the terminal database or the remote database based on the attribute characteristics of the target data corresponding to the data writing request after the data writing request is received.
Optionally, if the remote databases include a cloud database cluster and at least one edge database cluster, connecting at least two remote databases by:
connecting any two cloud data nodes through a network, respectively connecting all the edge data nodes in each edge database cluster through the network, and respectively connecting each cloud data node with at least one edge data node through the network;
the cloud data clusters are at least one remote database selected from at least two remote databases based on preset rules, the edge database clusters are obtained by dividing the remote databases except for cloud data nodes in the at least two remote databases based on regional information corresponding to the remote databases, the edge data nodes are the remote databases in the edge database clusters, and the number of the edge database clusters is at least one.
Optionally, the cloud data nodes and/or the edge data nodes corresponding to the region information of the end database are connected with the end database through a network.
Optionally, the target data is returned to the client that sends the data read request, and the returning unit 1002 is configured to:
determining a storage database storing target data, wherein the storage database is any one of an end database, an edge data node and a cloud data node;
determining an access database connected with a client side sending a data reading request, wherein the access database and the client side sending the data reading request are positioned at the same physical node;
if the access database sending the data reading request and the storage database are the same database, returning the target data from the storage database to the client sending the data reading request;
if the access database sending the data reading request is not the same database as the storage database, synchronizing the target data to the access database through the storage database, and returning the target data to the client sending the data reading request by using the access database.
Optionally, if the access database sending the data read request is not the same database as the storage database, the storage database is used to synchronize the target data to the access database, and the access database is used to return the target data to the client sending the data read request, and the return unit 1002 is configured to:
If the access database is an edge data node, the access database and the storage database are not the same database, and the access database and the storage database belong to the same edge database cluster, the target data is sent to the access database from the storage database, and the target data is returned to a client for sending a data reading request through the access database;
if the access database is an edge data node, the access database and the storage database are not the same database, and the access database and the storage database do not belong to the same edge database cluster, then the target data is sent to the cloud data node from the storage database, the target data is returned to the edge database cluster to which the access database belongs through the cloud data node, and the target data is sent to a client of a data reading request through the access database in the edge database cluster to which the access database belongs;
if the access database is a cloud data node and the access database and the storage database are not the same database, sending the target data from the storage database to the cloud data node, and sending the target data to a client of a data reading request through the access database;
if the access database is an end database and the access database and the storage database are not the same database, sending the target data from the storage database to the cloud data node, returning the target data to an edge database cluster connected with the access database through the cloud data node, returning the target data to the access database through the edge database cluster connected with the access database, and sending the target data to a client of a data reading request through the access database.
Optionally, the target data is stored in the end database or the remote database by:
responding to the received data writing request, and determining attribute characteristics of target data corresponding to the data writing request;
determining an end database or a far-end database matched with the attribute characteristics based on a preset storage rule;
writing the target data into the matched terminal database or the remote database, writing the identification generated based on the target data into the matched terminal database or the remote database, and sending the identification to the client, wherein the identification is a global variable.
Optionally, if the network link between the access database and the storage database fails, after responding to the received data writing request, the method further includes:
storing target data corresponding to the data writing request into an access database;
if the network link fault between the access database and the storage database is recovered to be normal, the method further comprises:
triggering the access database to store the target data in the storage database based on the attribute characteristics of the target data.
Optionally, in response to the received data reading request, the searching unit 1001 is configured to search the terminal database and the remote database for target data corresponding to the data reading request:
In response to the received data reading request, determining an identifier corresponding to the data reading request from a client side sending the data reading request;
searching the identifiers stored in the database, which are the same as the identifiers corresponding to the data reading requests, in the end database and the far-end database based on the identifiers corresponding to the data reading requests;
target data corresponding to the identification stored in the database is determined as target data corresponding to the data read request.
Based on the same inventive concept, referring to fig. 11, an embodiment of the present application provides an intelligent terminal, including: a memory 1101 for storing executable instructions; a processor 1102 for reading and executing executable instructions stored in a memory and performing any of the methods of the first aspect described above.
Based on the same inventive concept, an embodiment of the present application provides a computer-readable storage medium, which when executed by a processor, causes the processor to perform the method of any one of the above-described first aspects.
In summary, in the embodiment of the present application, a data storage method, a device, and a storage medium are provided, where the method includes: and searching target data corresponding to the data reading request in the terminal database and the remote database in response to the received data reading request, wherein the terminal database and the remote database are respectively different in corresponding region information, are connected through a network, and return the target data to a client for sending the data reading request, and the target data is stored in the terminal database or the remote database based on the attribute characteristics of the target data corresponding to the data writing request after receiving the data writing request, so that the distributed storage of the data is realized, the condition of data loss caused by wide data distribution region is avoided, and the efficiency and the accuracy of data query are improved.
It will be appreciated by those skilled in the art that embodiments of the present application may be provided as a method, system, or computer program product system. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product system embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.
The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program product systems according to the application. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
It will be apparent to those skilled in the art that various modifications and variations can be made to the present application without departing from the spirit or scope of the application. Thus, it is intended that the present application also include such modifications and alterations insofar as they come within the scope of the appended claims or the equivalents thereof.

Claims (10)

1. A method of reading and writing data, the method comprising:
if the remote databases comprise a cloud database cluster and at least one edge database cluster, connecting at least two of the remote databases by:
connecting any two cloud data nodes in the cloud database clusters through a network, respectively connecting all the edge data nodes in each edge database cluster through the network, and respectively connecting each cloud data node with at least one edge data node through the network;
the cloud data nodes are at least one far-end database selected from at least two far-end databases based on a preset rule, the edge database clusters are obtained by dividing the far-end databases except the cloud data nodes in the at least two far-end databases based on regional information corresponding to the far-end databases, the edge data nodes are all far-end databases in the edge database clusters, and the number of the edge database clusters is at least one group;
connecting cloud data nodes and/or edge data nodes corresponding to the region information of the end database with the end database through a network;
Searching target data corresponding to the data reading request in a terminal database and a remote database in response to the received data reading request, wherein the terminal database and the remote database are different in corresponding regional information, and the terminal database and the remote database are connected through a network;
returning the target data to a client side sending the data reading request;
the target data is stored in an end database or a far-end database based on the attribute characteristics of the target data corresponding to the data writing request after the data writing request is received, and the position information corresponding to the client side sending the data writing request is the same as the region information corresponding to the end database;
the returning the target data to the client side sending the data reading request comprises the following steps:
determining a storage database in which the target data is stored;
determining an access database connected with a client side sending the data reading request;
if the access database is the edge data node, the access database and the storage database are not the same database, and the access database and the storage database belong to the same edge database cluster, the target data is sent to the access database from the storage database, and the target data is returned to the client for sending the data reading request through the access database; or,
If the access database is the edge data node, the access database and the storage database are not the same database, and the access database and the storage database do not belong to the same edge database cluster, the target data is sent to the cloud data node from the storage database, the target data is returned to the edge database cluster to which the access database belongs through the cloud data node, and the target data is sent to the client of the data reading request through the access database in the edge database cluster to which the access database belongs; or,
if the access database is the cloud data node and the access database and the storage database are not the same database, the target data is sent to the cloud data node from the storage database, and the target data is sent to the client of the data reading request through the access database; or,
and if the access database is the end database and the access database and the storage database are not the same database, transmitting the target data from the storage database to the cloud data node, returning the target data to an edge database cluster connected with the access database through the cloud data node, and returning the target data to the access database through the edge database cluster connected with the access database, and transmitting the target data to the client of the data reading request through the access database.
2. The method of claim 1, wherein returning the target data to the client that sent the data read request further comprises:
if the target data is found in the terminal database, the target data is directly returned to the client side sending the data reading request;
and if the target data is found in the remote database, synchronizing the target data to the terminal database, and returning the target data to the client side sending the data reading request by using the terminal database.
3. The method of claim 2, wherein the end database is at the same physical node as the client of the data read request.
4. The method of claim 3, wherein the returning the target data to the client that sent the data read request further comprises:
if the access database sending the data reading request and the storage database are the same database, returning the target data from the storage database to the client sending the data reading request;
wherein the access database is at the same physical node as the client of the data read request.
5. The method of claim 3 or 4, wherein the target data is stored in an end database or a remote database by:
responding to the received data writing request, and determining attribute characteristics of target data corresponding to the data writing request;
determining the end database or the far-end database matched with the attribute features based on a preset storage rule;
writing the target data into the matched end database or the far-end database, writing the identification generated based on the target data into the matched end database or the far-end database, and sending the identification to a client, wherein the identification is a global variable.
6. The method of claim 5, wherein if a network link between the access database and the storage database fails, the responding to the received data write request further comprises:
storing target data corresponding to the data writing request into the access database;
if the network link between the access database and the storage database is recovered to be normal, the method further comprises:
Triggering the access database to store the target data into the storage database based on the attribute characteristics of the target data.
7. The method of claim 5, wherein in response to the received data read request, searching the end database and the remote database for target data corresponding to the data read request, comprises:
determining an identifier corresponding to the data reading request from a client side sending the data reading request in response to the received data reading request;
searching the identifiers stored in the database, which are the same as the identifiers corresponding to the data reading requests, in the terminal database and the remote database based on the identifiers corresponding to the data reading requests;
and determining the target data corresponding to the identification stored in the database as the target data corresponding to the data reading request.
8. A data reading and writing apparatus, comprising:
if the remote databases comprise a cloud database cluster and at least one edge database cluster, connecting at least two of the remote databases by:
connecting any two cloud data nodes in the cloud database clusters through a network, respectively connecting all the edge data nodes in each edge database cluster through the network, and respectively connecting each cloud data node with at least one edge data node through the network;
The cloud data nodes are at least one far-end database selected from at least two far-end databases based on a preset rule, the edge database clusters are obtained by dividing the far-end databases except the cloud data nodes in the at least two far-end databases based on regional information corresponding to the far-end databases, the edge data nodes are all far-end databases in the edge database clusters, and the number of the edge database clusters is at least one group;
connecting cloud data nodes and/or edge data nodes corresponding to the region information of the end database with the end database through a network;
the searching unit is used for responding to the received data reading request and searching target data corresponding to the data reading request in a terminal database and a remote database, wherein the terminal database and the remote database are respectively different in corresponding regional information, and the terminal database and the remote database are connected through a network;
the return unit is used for returning the target data to the client side which sends the data reading request;
the target data is stored in an end database or a far-end database based on the attribute characteristics of the target data corresponding to the data writing request after the data writing request is received, and the position information corresponding to the client side sending the data writing request is the same as the region information corresponding to the end database;
The return unit is specifically configured to determine a storage database in which the target data is stored;
determining an access database connected with a client side sending the data reading request;
if the access database is the edge data node, the access database and the storage database are not the same database, and the access database and the storage database belong to the same edge database cluster, the target data is sent to the access database from the storage database, and the target data is returned to the client for sending the data reading request through the access database; or,
if the access database is the edge data node, the access database and the storage database are not the same database, and the access database and the storage database do not belong to the same edge database cluster, the target data is sent to the cloud data node from the storage database, the target data is returned to the edge database cluster to which the access database belongs through the cloud data node, and the target data is sent to the client of the data reading request through the access database in the edge database cluster to which the access database belongs; or,
If the access database is the cloud data node and the access database and the storage database are not the same database, the target data is sent to the cloud data node from the storage database, and the target data is sent to the client of the data reading request through the access database; or,
and if the access database is the end database and the access database and the storage database are not the same database, transmitting the target data from the storage database to the cloud data node, returning the target data to an edge database cluster connected with the access database through the cloud data node, and returning the target data to the access database through the edge database cluster connected with the access database, and transmitting the target data to the client of the data reading request through the access database.
9. An intelligent terminal, characterized by comprising:
a memory for storing executable instructions;
a processor for reading and executing executable instructions stored in said memory to implement the method of any one of claims 1-7.
10. A computer readable storage medium, wherein instructions in the storage medium, when executed by a processor, enable the processor to perform the method of any one of claims 1-7.
CN202211191106.5A 2022-09-28 2022-09-28 Data reading and writing method, device and storage medium Active CN115481187B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202211191106.5A CN115481187B (en) 2022-09-28 2022-09-28 Data reading and writing method, device and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202211191106.5A CN115481187B (en) 2022-09-28 2022-09-28 Data reading and writing method, device and storage medium

Publications (2)

Publication Number Publication Date
CN115481187A CN115481187A (en) 2022-12-16
CN115481187B true CN115481187B (en) 2023-11-21

Family

ID=84393501

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202211191106.5A Active CN115481187B (en) 2022-09-28 2022-09-28 Data reading and writing method, device and storage medium

Country Status (1)

Country Link
CN (1) CN115481187B (en)

Citations (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106101055A (en) * 2016-04-29 2016-11-09 乐视控股(北京)有限公司 The data access method of a kind of multiple database and system thereof and proxy server
CN106341454A (en) * 2016-08-23 2017-01-18 世纪龙信息网络有限责任公司 Across-room multiple-active distributed database management system and across-room multiple-active distributed database management method
CN107704490A (en) * 2017-08-22 2018-02-16 贵州白山云科技有限公司 A kind of data processing method and device based on equity storage
CN108197159A (en) * 2017-12-11 2018-06-22 厦门集微科技有限公司 Digital independent, wiring method and device based on distributed file system
CN110837515A (en) * 2019-11-06 2020-02-25 北京天融信网络安全技术有限公司 Database-based data processing method and electronic equipment
CN111460240A (en) * 2020-04-13 2020-07-28 吉林亿联银行股份有限公司 Page turning data query method and device under cross-region multi-activity micro-service architecture
CN112799589A (en) * 2021-01-14 2021-05-14 新华三大数据技术有限公司 Data reading method and device
CN113190625A (en) * 2021-05-25 2021-07-30 中国工商银行股份有限公司 Request processing method, device, electronic equipment, medium and program product
CN113642036A (en) * 2021-07-07 2021-11-12 阿里巴巴新加坡控股有限公司 Data processing method, device and system
CN114595245A (en) * 2022-03-18 2022-06-07 上海哔哩哔哩科技有限公司 Data processing method and device

Patent Citations (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106101055A (en) * 2016-04-29 2016-11-09 乐视控股(北京)有限公司 The data access method of a kind of multiple database and system thereof and proxy server
CN106341454A (en) * 2016-08-23 2017-01-18 世纪龙信息网络有限责任公司 Across-room multiple-active distributed database management system and across-room multiple-active distributed database management method
CN107704490A (en) * 2017-08-22 2018-02-16 贵州白山云科技有限公司 A kind of data processing method and device based on equity storage
CN108197159A (en) * 2017-12-11 2018-06-22 厦门集微科技有限公司 Digital independent, wiring method and device based on distributed file system
CN110837515A (en) * 2019-11-06 2020-02-25 北京天融信网络安全技术有限公司 Database-based data processing method and electronic equipment
CN111460240A (en) * 2020-04-13 2020-07-28 吉林亿联银行股份有限公司 Page turning data query method and device under cross-region multi-activity micro-service architecture
CN112799589A (en) * 2021-01-14 2021-05-14 新华三大数据技术有限公司 Data reading method and device
CN113190625A (en) * 2021-05-25 2021-07-30 中国工商银行股份有限公司 Request processing method, device, electronic equipment, medium and program product
CN113642036A (en) * 2021-07-07 2021-11-12 阿里巴巴新加坡控股有限公司 Data processing method, device and system
CN114595245A (en) * 2022-03-18 2022-06-07 上海哔哩哔哩科技有限公司 Data processing method and device

Also Published As

Publication number Publication date
CN115481187A (en) 2022-12-16

Similar Documents

Publication Publication Date Title
US10002148B2 (en) Memory-aware joins based in a database cluster
CN103714097A (en) Method and device for accessing database
CN110377577B (en) Data synchronization method, device, system and computer readable storage medium
CN103873451A (en) Data synchronization method, equipment and system
CN103258036A (en) Distributed real-time search engine based on p2p
CN113987064A (en) Data processing method, system and equipment
CN110597852A (en) Data processing method, device, terminal and storage medium
CN116107828A (en) Main node selection method, distributed database and storage medium
CN108874950B (en) Data distribution storage method and device based on ER relationship
US11226986B2 (en) Data table partitioning management method and apparatus
CN107493309B (en) File writing method and device in distributed system
CN102724301B (en) Cloud database system and method and equipment for reading and writing cloud data
CN107908713B (en) Distributed dynamic rhododendron filtering system based on Redis cluster and filtering method thereof
CN107465706B (en) Distributed data object storage device based on wireless communication network
CN115481187B (en) Data reading and writing method, device and storage medium
CN111767282A (en) MongoDB-based storage system, data insertion method and storage medium
CN111552701A (en) Method for determining data consistency in distributed cluster and distributed data system
CN111352938A (en) Data processing method, computer device and storage medium
Pankowski Consistency and availability of Data in replicated NoSQL databases
CN107491270B (en) Resource access method and device of multi-control storage system
CN116186082A (en) Data summarizing method based on distribution, first server and electronic equipment
CN112579709B (en) Data table identification method and device, storage medium and electronic equipment
CN110765190B (en) Automatic database cluster capacity expansion method and device and electronic equipment
JP2021524085A (en) Message processing methods, devices and systems
JPWO2016067370A1 (en) Information processing apparatus, method, and program

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant