CN112835936A - Data processing method and device and service cluster - Google Patents
Data processing method and device and service cluster Download PDFInfo
- Publication number
- CN112835936A CN112835936A CN202110161980.3A CN202110161980A CN112835936A CN 112835936 A CN112835936 A CN 112835936A CN 202110161980 A CN202110161980 A CN 202110161980A CN 112835936 A CN112835936 A CN 112835936A
- Authority
- CN
- China
- Prior art keywords
- data
- identification information
- position identification
- service cluster
- location
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24552—Database cache management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computing Systems (AREA)
- Computational Linguistics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a data processing method, a data processing device and a service cluster, and relates to the technical field of computers. One embodiment of the method comprises: in response to receiving a data acquisition request, wherein the data acquisition request comprises: location information of the data; generating first position identification information according to the position information of the data; searching for effective second position identification information matched with the first position identification information in the cache region, wherein the effective second position identification information is used for indicating that corresponding data is not synchronized to a slave database included in the service cluster; and if the data is found, acquiring the data from a main database included in the service cluster according to the first position identification information. The implementation method can ensure consistency of data written into the service cluster and data acquired from the service cluster.
Description
Technical Field
The present invention relates to the field of computer technologies, and in particular, to a data processing method, an apparatus, and a service cluster.
Background
To improve the efficiency of data storage and retrieval, many enterprises use the cooperation of master and slave databases in a service cluster to accomplish data access. That is, by storing data into a master database and storing the data stored in the master database to a plurality of slave databases in a distributed manner, so that a large number of users read or acquire data through the plurality of slave databases in a distributed manner, it is possible to relieve the access pressure of the master database.
However, during the writing of data to the master database, the data has not yet been written to the slave database. Then, during the process of writing the data into the master database, the user cannot obtain the data from the database, resulting in poor consistency of the data.
Disclosure of Invention
In view of this, embodiments of the present invention provide a data processing method, an apparatus, and a service cluster, which can ensure consistency between data written into the service cluster and data acquired from the service cluster.
To achieve the above object, according to an aspect of the embodiments of the present invention, there is provided a data processing method applied to a service cluster, including:
in response to receiving a data acquisition request, wherein the data acquisition request comprises: location information of the data;
generating first position identification information according to the position information of the data;
searching a cache region for effective second location identification information matched with the first location identification information, wherein the effective second location identification information is used for indicating that corresponding data is not synchronized to a slave database included in the service cluster;
and if the data is found, acquiring data from a main database included in the service cluster according to the first position identification information.
Preferably, the data processing method further comprises:
and if the cache region does not search for the effective second position identification information matched with the first position identification information, acquiring data from a slave database included in the service cluster according to the first position identification information.
Preferably, the data processing method further comprises:
when data is written into a main database included in the service cluster, generating second position identification information for the written data according to the written position information of the data;
caching the second position identification information into the cache region, and setting corresponding effective duration for the second position identification information;
when the caching duration of the second position identification information in the caching area does not reach the effective duration, determining that the second position identification information is effective;
and when the caching duration of the second position identification information in the caching area reaches the effective duration, determining that the second position identification information is invalid.
Preferably, the first and second electrodes are formed of a metal,
the valid period is determined by a time point at which the master database writes the data and a time point at which the data written in the master database is synchronized to the slave database.
Preferably, the data processing method further comprises:
counting the access conditions of a plurality of second position identification information;
according to the statistical result, selecting at least one target second position identification information from the plurality of second position identification information;
and caching the data corresponding to the at least one target second position identification information synchronously completed from the database into the cache region.
Preferably, selecting at least one target second location identification information comprises:
and selecting the second position identification information with the access times not lower than a preset access time threshold as the target second position identification information based on the access times received by each second position identification information.
Preferably, selecting at least one target second location identification information comprises:
and selecting the second position identification information meeting the caching condition as target second position identification information based on the preset caching condition.
Preferably, the data processing method further comprises:
and when the first position identification information is matched with any one of the target second position identification information and the target second position identification information matched with the first position identification information is identified as invalid, acquiring data corresponding to the first position identification information from the cache region.
In a second aspect, an embodiment of the present invention provides a data processing apparatus, applied to a service cluster, including: an interaction unit and a processing unit, wherein,
the interaction unit is configured to receive a data acquisition request, where the data acquisition request includes: location information of the data;
the processing unit is used for generating first position identification information according to the position information of the data; searching a cache region for effective second location identification information matched with the first location identification information, wherein the effective second location identification information is used for indicating that corresponding data is not synchronized to a slave database included in the service cluster;
the interaction unit is further configured to, if the processing unit finds valid second location identifier information matching the first location identifier information, obtain data from a master database included in the service cluster according to the first location identifier information.
In a third aspect, an embodiment of the present invention provides a service cluster, including: the master database, the slave database, the buffer and the data processing device provided by the above embodiments, wherein,
the main database is used for writing data, and when the data processing device searches for effective second position identification information matched with the first position identification information in a cache region, the main database provides the data to the data processing device;
the slave database is used for synchronously storing the data written in the master database, and when the data processing device does not search the effective second position identification information matched with the first position identification information in the cache region, providing the data to the data processing device;
the cache region is configured to cache valid second location identifier information of the data written by the master database, where the valid second location identifier information is used to indicate that the corresponding data is not synchronized to a slave database included in the service cluster.
One embodiment of the above invention has the following advantages or benefits: according to the data acquisition method and device, the data corresponding to the second position identification information is effectively indicated to be not synchronized to the slave database included in the service cluster, the second position identification information is invalid to indicate that the data corresponding to the second position identification information is synchronized to the slave database included in the service cluster, the first position identification information can be generated through the position information of the data included in the received data acquisition request, whether the effective second position identification information matched with the first position identification information exists or not is searched in the cache region, if the effective second position identification information is matched with the first position identification information, the fact that the data required to be acquired by the data acquisition request are only stored in the master database included in the service cluster and are not synchronized to the slave database is shown, the data can be acquired from the master database included in the service cluster according to the first position identification information, and the acquired data are guaranteed to be consistent with the data stored in the service cluster.
Further effects of the above-mentioned non-conventional alternatives will be described below in connection with the embodiments.
Drawings
The drawings are included to provide a better understanding of the invention and are not to be construed as unduly limiting the invention. Wherein:
FIG. 1 is a schematic diagram of a service cluster according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of a main flow of a data processing method according to an embodiment of the present invention;
fig. 3 is a schematic diagram illustrating a main flow of validity/invalidity of second location identity information according to an embodiment of the present invention;
FIG. 4 is a diagram illustrating a main flow of data caching to a cache according to an embodiment of the present invention;
FIG. 5 is a schematic diagram of the main elements of a data processing apparatus according to an embodiment of the present invention;
FIG. 6 is a schematic diagram of a service cluster according to an embodiment of the present invention;
FIG. 7 is an exemplary system architecture diagram in which embodiments of the present invention may be employed;
fig. 8 is a schematic structural diagram of a computer system suitable for implementing a terminal device or a server according to an embodiment of the present invention.
Detailed Description
Exemplary embodiments of the present invention are described below with reference to the accompanying drawings, in which various details of embodiments of the invention are included to assist understanding, and which are to be considered as merely exemplary. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the invention. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.
The service cluster on which the embodiment of the present invention is based is shown in fig. 1. The service cluster may include: a master database 101, a plurality of slave databases 102 connected to the master database 101, wherein the plurality of slave databases 102 can synchronize data in the master database 101 to provide required data for external systems, thereby reducing communication pressure of the master database 101. In addition, a buffer 103 independent of the master database 101 and the plurality of slave databases 102 is provided, and the buffer may buffer some data frequently called by an external system. The data cached in the cache region 103 needs to be cleaned up some failed data regularly. During the process of writing data in the master database 101, the data being written in the master database 101 is not synchronized to the slave database 102 and the buffer 103, and if the external system obtains the data from the slave database 102 or the buffer 103, the data read or obtained by the external system will be dirty. The master and slave databases are synchronized with a time delay during which reading of the slave database may result in inconsistent data being read by the external system. It should be understood that the number of master databases, slave databases, and buffers in fig. 1 is merely illustrative. There may be any number of master databases, slave databases, and buffers, as the implementation requires.
In order to ensure that the external system can acquire accurate data from the service cluster and ensure that the data acquired or read by the external system is consistent with the data written or stored by the service cluster, it is necessary to control whether the external system acquires the data from the master database, or acquires the data from the buffer. Fig. 2 is a data processing method applied to a service cluster according to an embodiment of the present invention, and as shown in fig. 2, the data processing method may include the following steps:
step S201: in response to receiving a data acquisition request, wherein the data acquisition request comprises: location information of the data;
the position information of the data refers to a database, a data table, a corresponding primary key of the data in the data table, and the like, in which the data is written, and the database and the data table in which the data is located can be located through the position information, and the accurate position of the data table in which the data is located can be found through the primary key.
Step S202: generating first position identification information according to the position information of the data;
the specific implementation manner of this step can be two.
The first specific implementation manner: and arranging the codes of the database, the codes of the data table and the corresponding main keys of the data in the data table according to a set sequence to obtain first position identification information. For example, if the database code is 001, the data table code is a101, and the data table corresponds to a primary key of 110011, and the set arrangement order is the database code, the data table code, and the data table corresponding to the primary key, the first location identity information is 001a 101110011.
The second specific implementation manner is as follows: and obtaining the first position identification information according to the set calculation mode by the codes of the database, the codes of the data table and the corresponding main keys of the data in the data table. Such as: the code of the database is 001, the code of the data table is a101, and the corresponding primary key of the data in the data table is 110011, the set calculation mode is that the code of the database, the code of the data table and the corresponding primary key of the data in the data table are accumulated (when the code of the non-number is involved, the code of the non-number digit is added into the first position identification information, the position of the code of the non-number digit is consistent with the position of the code in the original code), the first position identification information is 001+ a101+110011 ═ 110a113, because the a comes from the original code as a101, the position of the a in the first position identification information is the fourth bit from right to left, and is also the fourth bit from right to left in the code of the data table.
Step S203: searching for effective second position identification information matched with the first position identification information in the cache region, wherein the effective second position identification information is used for indicating that corresponding data is not synchronized to a slave database included in the service cluster; if the data is found, executing step S204; if not, executing step S205;
the second location identification information is stored in the buffer and is determined according to the location of the data stored in the main database. The second position identification information is generated in a manner consistent with the first position identification information.
The first location identification information is matched with the effective second location identification information, which means that the first location identification information is the same as the effective second location identification information.
Step S204: acquiring data from a main database included in the service cluster according to the first position identification information, and ending the current process;
when the second position identification information matched with the first position identification information is found, the data is stored in the master database, but the data is not synchronized to the slave database. The data corresponding to the first location identity information is obtained from the master database.
Step S205: data is obtained from a slave database included in the service cluster based on the first location identity information.
When the effective second position identification information matched with the first position identification information is not found, the data is synchronized to the slave database. The data corresponding to the first location identity information is obtained from the database.
In the embodiment shown in fig. 2, if the second location identification information is valid to indicate that the corresponding data is not synchronized to the slave database included in the service cluster, and the second location identification information is invalid to indicate that the corresponding data is synchronized to the slave database included in the service cluster, the first location identification information may be generated by the location information of the data included in the received data obtaining request, and if there is valid second location identification information matching the first location identification information in the cache, it indicates that the data obtaining request wants to obtain is only stored in the master database in the service cluster, and if it is not synchronized to the slave database, the data may be obtained from the master database included in the service cluster according to the first location identification information, so as to ensure that the obtained data is consistent with the data stored in the service cluster.
In the embodiment of the invention, in order to accurately monitor whether the data written into the master database is synchronized into the slave database, second position identification information is introduced, and the storage position of the data is monitored through whether the second position identification information is effective. As shown in fig. 3, the data processing method may further include the steps of:
step S301: when data is written into a main database included in the service cluster, generating second position identification information for the written data according to the position information of the written data;
the second position identification information is generated in a manner consistent with that of the first position identification information, so that the first position identification information and the second position identification information can be conveniently matched.
Step S302: caching the second position identification information into a cache region, and setting corresponding effective duration for the second position identification information; when the caching duration of the second location identity information in the caching area does not reach the effective duration, executing step S303; when the buffering duration of the second location identity information in the buffering area reaches the effective duration, executing step S304;
the valid period is determined by a time point at which the master database writes data and a time point at which the data written to the master database is synchronized to the slave database. For example, when a data is written into the master database at 11 o 'clock 30 min 22 sec, and the data is synchronized to the slave database at 11 o' clock 30 min 22 sec 225 msec, the valid duration corresponding to the second location identity information corresponding to the data is 225 msec. For another example, the time point of writing the other data into the master database is 11 o 'clock 30 min 22 sec, the time point of synchronizing the other data to the completion of the slave database is 11 o' clock 30 min 22 sec 500 msec, and the valid duration corresponding to the second location identity information corresponding to the other data is 500 msec.
In addition, when the second location identification information related to a certain database and a certain data table is generated for the first time, the valid duration corresponding to the setting may also be: recording the time point of writing data in the master database, setting second position identification information, writing the synchronized time point into the cache region after receiving the data synchronization corresponding to the second position identification information to the slave database, and setting the second position identification information invalid correspondingly. And the difference value between the synchronous completion time point and the time point of recording the data written in the main database is the effective time length. And when other second position identification information related to the certain database and the certain data table is generated again in the following, directly setting the effective duration for the other second position identification information.
Step S303: determining that the second position identification information is valid, and ending the current process;
step S304: it is determined that the second location identity information is invalid.
In an embodiment of the present invention, in order to obtain data more quickly and reduce overhead for the master database and the slave database, as shown in fig. 4, the data processing method may further include the following steps:
step S401: counting the access conditions of the plurality of second position identification information;
the access condition may refer to the access frequency or the access times of the data within a set time.
Step S402: selecting at least one target second position identification information from the plurality of second position identification information according to the statistical result;
there are two possible implementations of this step:
a first implementation of selecting at least one target second location identity information:
and selecting the second position identification information with the access times not lower than a preset access time threshold as the target second position identification information based on the access times received by each second position identification information. The preset access time threshold value can be set correspondingly according to actual requirements.
A second implementation manner of selecting at least one target second location identity information:
and selecting the second position identification information meeting the caching condition as target second position identification information based on the preset caching condition. Wherein, the buffer condition may be the second location identification information of the code containing a certain data table.
Step S403: and caching data corresponding to the at least one target second position identification information synchronously completed from the database into a cache region.
By caching the data in the cache region, the overhead of the master database and the slave database can be effectively reduced, and the pressure of the master database and the slave database is reduced. Meanwhile, the data transmission efficiency can be effectively improved.
In an embodiment of the present invention, the data processing method may further include: and when the first position identification information is matched with any target second position identification information and the target second position identification information matched with the first position identification information is identified as invalid, acquiring data corresponding to the first position identification information from the cache region. The second location identification information is invalid, which indicates that the data corresponding to the second location identification information is synchronized to the cache region, and then the data is acquired from the cache region, so that the consistency of the data can be ensured.
As shown in fig. 5, an embodiment of the present invention provides a data processing apparatus 500, where the data processing apparatus 500 is applied to a service cluster, and the data processing apparatus 500 may include: an interaction unit 501 and a processing unit 502, wherein,
an interaction unit 501, configured to receive a data obtaining request, where the data obtaining request includes: location information of the data;
a processing unit 502, configured to generate first location identification information according to location information of data; searching for effective second position identification information matched with the first position identification information in the cache region, wherein the effective second position identification information is used for indicating that corresponding data is not synchronized to a slave database included in the service cluster;
the interacting unit 501 is further configured to, if the processing unit 502 finds valid second location identity information matching the first location identity information, obtain data from a master database included in the service cluster according to the first location identity information.
In this embodiment of the present invention, the interaction unit 501 is further configured to, if the processing unit 502 does not find valid second location identity information matching the first location identity information, obtain data from a master database included in the service cluster according to the first location identity information.
In this embodiment of the present invention, the processing unit 502 is further configured to, when data is written to a master database included in the service cluster, generate second location identification information for the written data according to location information of the written data; caching the second position identification information into the cache region, and setting corresponding effective duration for the second position identification information; when the caching duration of the second position identification information in the caching area does not reach the effective duration, determining that the second position identification information is effective; and when the caching duration of the second position identification information in the caching area reaches the effective duration, determining that the second position identification information is invalid.
In the embodiment of the present invention, the effective duration set by the processing unit 502 for the second location identity information is determined by the time point of writing data in the master database and the time point of synchronizing the data written in the master database to the slave database.
In this embodiment of the present invention, the processing unit 502 is further configured to count access conditions of a plurality of second location identity information; selecting at least one target second position identification information from the plurality of second position identification information according to the statistical result; and caching data corresponding to the at least one target second position identification information synchronously completed from the database into a cache region.
In this embodiment of the present invention, the processing unit 502 is configured to select, as the target second location identity information, the second location identity information having an access count not lower than a preset access count threshold based on the access count received by each second location identity information.
In this embodiment of the present invention, the processing unit 502 is configured to select, based on a preset caching condition, second location identity information that meets the caching condition as target second location identity information.
In this embodiment of the present invention, the interaction unit 501 is further configured to obtain data corresponding to the first location identity information from the cache when the first location identity information matches any of the target second location identity information and the target second location identity information matching the first location identity information is invalid. As shown in fig. 6, an embodiment of the present invention provides a service cluster 600, where the service cluster 600 may include: the master database 601, the slave database 602, the buffer 603 and the data processing apparatus 604 according to any of the above embodiments, wherein,
a main database 601 for writing data, and providing data to the data processing apparatus when the data processing apparatus 604 searches the cache region for effective second location identification information matching the first location identification information;
the slave database 602 is used for synchronously storing the data written in the master database, and when the data processing device 604 does not search the effective second position identification information matched with the first position identification information in the cache region, the data is provided for the data processing device 604;
the buffer 603 is configured to buffer valid second location identification information of the data written by the master database 601, where the valid second location identification information is used to indicate that the corresponding data is not synchronized to the slave database included in the service cluster.
Fig. 7 shows an exemplary system architecture 700 of a data processing method or data processing apparatus to which embodiments of the present invention may be applied.
As shown in fig. 7, the system architecture 700 may include end devices 701, 702, 703, a network 704, an external system server 705, a data processing server 706, and a service cluster 707. The service cluster 707 includes a master database 7071, a plurality of slave databases 7072, and a buffer 7073. Network 704 is the medium used to provide communications links between terminal devices 701, 702, 703, external system server 705, and data processing server 706 and service cluster 707. Network 704 may include various connection types, such as wired, wireless communication links, or fiber optic cables, to name a few.
The user may use the terminal devices 701, 702, 703 to interact with the data processing server 706 and the service cluster 707 over the network 704 to receive the master database 7071 in the service cluster 707 or to transmit data from the database 7072 or the buffer 7073 or to the master database 7071 in the service cluster 707, and so on. The external system server 705 may also interact with the data processing server 706 and the service cluster 707 to receive data from the master database 7071 in the service cluster 707 or to transmit data from the database 7072 or the buffer 7073 or to the master database 7071 in the service cluster 707, and so on. The terminal devices 701, 702, 703 may have installed thereon various communication client applications, such as a shopping-type application, a web browser application, a search-type application (just an example).
The terminal devices 701, 702, 703 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop portable computers, desktop computers, and the like.
It should be noted that the data processing method provided by the embodiment of the present invention is generally executed by the data processing server 706, and accordingly, the data processing apparatus is generally disposed in the data processing server 706.
It should be understood that the number of terminal devices, networks, external system servers, data processing servers, and service clusters in fig. 7 are merely illustrative. There may be any number of terminal devices, networks, external system servers, data processing servers, and service clusters, as desired for implementation.
Referring now to FIG. 8, shown is a block diagram of a computer system 800 suitable for use with a terminal device or server implementing an embodiment of the present invention. The terminal device shown in fig. 8 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present invention.
As shown in fig. 8, the computer system 800 includes a Central Processing Unit (CPU)801 that can perform various appropriate actions and processes in accordance with a program stored in a Read Only Memory (ROM)802 or a program loaded from a storage section 808 into a Random Access Memory (RAM) 803. In the RAM 803, various programs and data necessary for the operation of the system 800 are also stored. The CPU 801, ROM 802, and RAM 803 are connected to each other via a bus 804. An input/output (I/O) interface 805 is also connected to bus 804.
The following components are connected to the I/O interface 805: an input portion 806 including a keyboard, a mouse, and the like; an output section 807 including a signal such as a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and the like, and a speaker; a storage portion 808 including a hard disk and the like; and a communication section 809 including a network interface card such as a LAN card, a modem, or the like. The communication section 809 performs communication processing via a network such as the internet. A drive 810 is also connected to the I/O interface 805 as necessary. A removable medium 811 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on the drive 810 as necessary, so that a computer program read out therefrom is mounted on the storage section 808 as necessary.
In particular, according to the embodiments of the present disclosure, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the method illustrated in the flow chart. In such an embodiment, the computer program can be downloaded and installed from a network through the communication section 809 and/or installed from the removable medium 811. The computer program executes the above-described functions defined in the system of the present invention when executed by the Central Processing Unit (CPU) 801.
It should be noted that the computer readable medium shown in the present invention can be a computer readable signal medium or a computer readable storage medium or any combination of the two. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present invention, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In the present invention, however, a computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, fiber optic cable, RF, etc., or any suitable combination of the foregoing.
The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The units described in the embodiments of the present invention may be implemented by software or hardware. The described units may also be provided in a processor, and may be described as: a processor includes an interaction unit and a processing unit. Where the names of these units do not in some cases constitute a limitation on the unit itself, for example, an interactive unit may also be described as a "unit that receives a data acquisition request".
As another aspect, the present invention also provides a computer-readable medium that may be contained in the apparatus described in the above embodiments; or may be separate and not incorporated into the device. The computer readable medium carries one or more programs which, when executed by a device, cause the device to comprise: in response to receiving a data acquisition request, wherein the data acquisition request comprises: location information of the data; generating first position identification information according to the position information of the data; searching for effective second position identification information matched with the first position identification information in the cache region, wherein the effective second position identification information is used for indicating that corresponding data is not synchronized to a slave database included in the service cluster; and if the data is found, acquiring the data from a main database included in the service cluster according to the first position identification information.
According to the technical scheme of the embodiment of the invention, the corresponding data is not synchronized to the slave database included in the service cluster through the effective indication of the second position identification information, the corresponding data is synchronized to the slave database included in the service cluster through the ineffective indication of the second position identification information, the first location identification information may be generated by the location information of the data included in the received data acquisition request, by looking up in the buffer whether there is valid second location identity information matching the first location identity information, if the data is matched with the service cluster, the data acquisition request only needs to acquire the data which is only stored in the master database in the service cluster and is not synchronized to the slave database, data may be obtained from a master database included in the service cluster according to the first location identity information to ensure that the obtained data is consistent with the data stored in the service cluster.
The above-described embodiments should not be construed as limiting the scope of the invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions can occur, depending on design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (12)
1. A data processing method is applied to a service cluster and comprises the following steps:
in response to receiving a data acquisition request, wherein the data acquisition request comprises: location information of the data;
generating first position identification information according to the position information of the data;
searching a cache region for effective second location identification information matched with the first location identification information, wherein the effective second location identification information is used for indicating that corresponding data is not synchronized to a slave database included in the service cluster;
and if the data is found, acquiring data from a main database included in the service cluster according to the first position identification information.
2. The data processing method of claim 1, further comprising:
and if the cache region does not search for the effective second position identification information matched with the first position identification information, acquiring data from a slave database included in the service cluster according to the first position identification information.
3. The data processing method of claim 1, further comprising:
when data is written into a main database included in the service cluster, generating second position identification information for the written data according to the written position information of the data;
caching the second position identification information into the cache region, and setting corresponding effective duration for the second position identification information;
when the caching duration of the second position identification information in the caching area does not reach the effective duration, determining that the second position identification information is effective;
and when the caching duration of the second position identification information in the caching area reaches the effective duration, determining that the second position identification information is invalid.
4. The data processing method of claim 3,
the valid period is determined by a time point at which the master database writes the data and a time point at which the data written in the master database is synchronized to the slave database.
5. The data processing method of claim 1, further comprising:
counting the access conditions of a plurality of second position identification information;
according to the statistical result, selecting at least one target second position identification information from the plurality of second position identification information;
and caching the data corresponding to the at least one target second position identification information synchronously completed from the database into the cache region.
6. The data processing method of claim 5, wherein selecting at least one target second location identity information comprises:
and selecting the second position identification information with the access times not lower than a preset access time threshold as the target second position identification information based on the access times received by each second position identification information.
7. The data processing method of claim 5, wherein selecting at least one target second location identity information comprises:
and selecting the second position identification information meeting the caching condition as target second position identification information based on the preset caching condition.
8. The data processing method of claim 5, further comprising:
and when the first position identification information is matched with any one of the target second position identification information and the target second position identification information matched with the first position identification information is identified as invalid, acquiring data corresponding to the first position identification information from the cache region.
9. A data processing apparatus, applied to a service cluster, comprising: an interaction unit and a processing unit, wherein,
the interaction unit is configured to receive a data acquisition request, where the data acquisition request includes: location information of the data;
the processing unit is used for generating first position identification information according to the position information of the data; searching a cache region for effective second location identification information matched with the first location identification information, wherein the effective second location identification information is used for indicating that corresponding data is not synchronized to a slave database included in the service cluster;
the interaction unit is further configured to, if the processing unit finds valid second location identifier information matching the first location identifier information, obtain data from a master database included in the service cluster according to the first location identifier information.
10. A service cluster, comprising: master database, slave database, buffer and data processing apparatus according to claim 9, wherein,
the main database is used for writing data, and when the data processing device searches for effective second position identification information matched with the first position identification information in a cache region, the main database provides the data to the data processing device;
the slave database is used for synchronously storing the data written in the master database, and when the data processing device does not search the effective second position identification information matched with the first position identification information in the cache region, providing the data to the data processing device;
the cache region is configured to cache valid second location identifier information of the data written by the master database, where the valid second location identifier information is used to indicate that the corresponding data is not synchronized to a slave database included in the service cluster.
11. An electronic device, comprising:
one or more processors;
a storage device for storing one or more programs,
when executed by the one or more processors, cause the one or more processors to implement the method of any one of claims 1-8.
12. A computer-readable medium, on which a computer program is stored, which, when being executed by a processor, carries out the method according to any one of claims 1-8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110161980.3A CN112835936A (en) | 2021-02-05 | 2021-02-05 | Data processing method and device and service cluster |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110161980.3A CN112835936A (en) | 2021-02-05 | 2021-02-05 | Data processing method and device and service cluster |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112835936A true CN112835936A (en) | 2021-05-25 |
Family
ID=75932430
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110161980.3A Pending CN112835936A (en) | 2021-02-05 | 2021-02-05 | Data processing method and device and service cluster |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112835936A (en) |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107656937A (en) * | 2016-07-26 | 2018-02-02 | 北京京东尚科信息技术有限公司 | Method and apparatus for realizing read-write data consistency |
| CN111241198A (en) * | 2020-01-02 | 2020-06-05 | 广州虎牙科技有限公司 | Data synchronization method and device and data processing equipment |
| CN111258822A (en) * | 2020-01-15 | 2020-06-09 | 广州虎牙科技有限公司 | Data processing method, server and computer readable storage medium |
| US20200204624A1 (en) * | 2017-07-28 | 2020-06-25 | Beijing Jingdong Shangke Information Technology Co., Ltd. | Data processing system, method and apparatus |
-
2021
- 2021-02-05 CN CN202110161980.3A patent/CN112835936A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107656937A (en) * | 2016-07-26 | 2018-02-02 | 北京京东尚科信息技术有限公司 | Method and apparatus for realizing read-write data consistency |
| US20200204624A1 (en) * | 2017-07-28 | 2020-06-25 | Beijing Jingdong Shangke Information Technology Co., Ltd. | Data processing system, method and apparatus |
| CN111241198A (en) * | 2020-01-02 | 2020-06-05 | 广州虎牙科技有限公司 | Data synchronization method and device and data processing equipment |
| CN111258822A (en) * | 2020-01-15 | 2020-06-09 | 广州虎牙科技有限公司 | Data processing method, server and computer readable storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109657174B (en) | Method and device for updating data | |
| CN109388626B (en) | Method and apparatus for assigning numbers to services | |
| US11599425B2 (en) | Method, electronic device and computer program product for storage management | |
| CN111737564A (en) | Information query method, device, equipment and medium | |
| CN113760982A (en) | Data processing method and device | |
| CN112835936A (en) | Data processing method and device and service cluster | |
| CN112148705A (en) | Data migration method and device | |
| CN113076343B (en) | Data query method, device, equipment and storage medium | |
| CN115640280A (en) | Data migration method and device | |
| CN115203210A (en) | Hash table processing method, device and equipment and computer readable storage medium | |
| CN113742376A (en) | Data synchronization method, first server and data synchronization system | |
| CN112711572B (en) | Online capacity expansion method and device suitable for database and table division | |
| CN109087097B (en) | Method and device for updating same identifier of chain code | |
| CN113760469A (en) | Distributed computing method and device | |
| CN114647686A (en) | Data processing method, device, equipment, medium and product | |
| CN108984431B (en) | Method and apparatus for flushing stale caches | |
| CN111125112A (en) | Data processing method and device, electronic equipment and computer readable storage medium | |
| CN112214500A (en) | Data comparison method and device, electronic equipment and storage medium | |
| CN113760860B (en) | Data reading method and device | |
| CN111695059A (en) | Information view list processing method and device and display equipment | |
| CN114500438B (en) | File sharing method and device, electronic equipment and storage medium | |
| CN113778909B (en) | Method and device for caching data | |
| EP4131017A2 (en) | Distributed data storage | |
| CN114995764A (en) | Data storage method and device based on stream computing | |
| CN113779082A (en) | Method and device for updating data |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20220318 Address after: 100176 Room 201, 2 / F, block C, 18 Kechuang 11th Street, Beijing Economic and Technological Development Zone, Daxing District, Beijing Applicant after: BEIJING JINGDONG CENTURY TRADING Co.,Ltd. Address before: 100176 room 703, 7th floor, building 1, yard 18, Kechuang 11th Street, Beijing Economic and Technological Development Zone, Beijing Applicant before: Beijing Dianzhi Technology Co.,Ltd. |
|
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20210525 |